[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:06:11 dist-test-slave-5p8w krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:06:11 dist-test-slave-5p8w krb5kdc[395](info): set up 2 sockets
Jun 19 01:06:11 dist-test-slave-5p8w krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 01:06:12.430969 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:06:12 dist-test-slave-5p8w krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:06:12.613620 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36863
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:12.963594 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:12.963919 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:12.964099 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:12.971618 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:12.971751 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:12.971850 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:12.971961 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:12.972021 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:06:12.982690 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:12.984134 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:12.985744 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:12.998780 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:12.998800 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:13.001332 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:13.001426 408 server_base.cc:1034] running on GCE node
I20260619 01:06:13.002449 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:13.004130 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:13.005385 408 hybrid_clock.cc:648] HybridClock initialized: now 1781831173005268 us; error 110 us; skew 500 ppm
I20260619 01:06:13.005721 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:13 dist-test-slave-5p8w krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:13.009019 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:06:13.010847 408 webserver.cc:460] Webserver started at http://127.0.97.62:35137/ using document root <none> and password file <none>
I20260619 01:06:13.011648 408 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:13.011803 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:13.012138 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:13.015461 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "8797e7309c3f4c4ebbc365e453f941b5"
format_stamp: "Formatted at 2026-06-19 01:06:13 on dist-test-slave-5p8w"
server_key: "ee87eefc45f2877663e72dddf4984534"
server_key_iv: "96d3d77ee2aac3a4a0389377192d6690"
server_key_version: "encryptionkey@0"
I20260619 01:06:13.016306 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "8797e7309c3f4c4ebbc365e453f941b5"
format_stamp: "Formatted at 2026-06-19 01:06:13 on dist-test-slave-5p8w"
server_key: "ee87eefc45f2877663e72dddf4984534"
server_key_iv: "96d3d77ee2aac3a4a0389377192d6690"
server_key_version: "encryptionkey@0"
I20260619 01:06:13.022140 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260619 01:06:13.026360 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:13.027925 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:06:13.028162 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "8797e7309c3f4c4ebbc365e453f941b5"
format_stamp: "Formatted at 2026-06-19 01:06:13 on dist-test-slave-5p8w"
server_key: "ee87eefc45f2877663e72dddf4984534"
server_key_iv: "96d3d77ee2aac3a4a0389377192d6690"
server_key_version: "encryptionkey@0"
I20260619 01:06:13.028393 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:13.061478 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:13.066402 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:13.066744 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:13.087924 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36863
I20260619 01:06:13.087946 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36863 every 8 connection(s)
I20260619 01:06:13.089586 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:13.093624 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:13.093604 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 408
I20260619 01:06:13.093923 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:13.095876 388 external_mini_cluster.cc:1428] Setting key c4adc4d66fd8ad5c49cd07f7deb26f1e
I20260619 01:06:13.106118 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5: Bootstrap starting.
I20260619 01:06:13.109418 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:13.110597 475 log.cc:824] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:13.114014 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5: No bootstrap required, opened a new log
I20260619 01:06:13.122181 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8797e7309c3f4c4ebbc365e453f941b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36863 } }
I20260619 01:06:13.122627 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:13.122745 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8797e7309c3f4c4ebbc365e453f941b5, State: Initialized, Role: FOLLOWER
I20260619 01:06:13.123464 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8797e7309c3f4c4ebbc365e453f941b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36863 } }
I20260619 01:06:13.123733 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:13.123888 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:13.124073 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 01:06:13 dist-test-slave-5p8w krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:13.127303 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8797e7309c3f4c4ebbc365e453f941b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36863 } }
I20260619 01:06:13.127889 475 leader_election.cc:302] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8797e7309c3f4c4ebbc365e453f941b5; no voters:
I20260619 01:06:13.128618 475 leader_election.cc:288] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:13.128751 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:13.134166 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [term 1 LEADER]: Becoming Leader. State: Replica: 8797e7309c3f4c4ebbc365e453f941b5, State: Running, Role: LEADER
I20260619 01:06:13.134908 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8797e7309c3f4c4ebbc365e453f941b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36863 } }
I20260619 01:06:13.135378 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:13.138031 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8797e7309c3f4c4ebbc365e453f941b5. Latest consensus state: current_term: 1 leader_uuid: "8797e7309c3f4c4ebbc365e453f941b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8797e7309c3f4c4ebbc365e453f941b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36863 } } }
I20260619 01:06:13.138435 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:13.139073 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8797e7309c3f4c4ebbc365e453f941b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8797e7309c3f4c4ebbc365e453f941b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36863 } } }
I20260619 01:06:13.139374 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:13.148000 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:13.151714 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:13.100001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.0.97.62:36863)
0619 01:06:13.100943 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:13.100963 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:13.102176 (+ 1213us) server_negotiation.cc:407] Connection header received
0619 01:06:13.112323 (+ 10147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:13.112350 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:13.112807 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:13.113256 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:13.116739 (+ 3483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:13.118279 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:13.120208 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:13.121012 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:13.121276 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:13.128570 (+ 7294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:13.128585 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:13.128608 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:13.128662 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:13.138201 (+ 9539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:13.140464 (+ 2263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:13.140482 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:13.140495 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:13.140919 (+ 424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:13.142312 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:13.142319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:13.142322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:13.143990 (+ 1668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:13.149061 (+ 5071us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:13.149442 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 01:06:13.150010 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":385,"threads_started":1}
I20260619 01:06:13.154021 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:13.165557 485 catalog_manager.cc:1347] Generated new cluster ID: 03a46afd18bf4cbd81ce65dcd429e3ef
I20260619 01:06:13.165704 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:13.189981 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:13.192536 485 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:13.201889 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8797e7309c3f4c4ebbc365e453f941b5: Generated new TSK 0
I20260619 01:06:13.202944 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:06:13.253726 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36863
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:13.606590 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:13.606928 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:13.607108 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:13.614630 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:13.614794 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:13.614965 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:06:13.624958 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:13.626794 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:13.628191 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:13.642717 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:13.642977 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:13.643225 501 server_base.cc:1034] running on GCE node
W20260619 01:06:13.643455 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:13.644170 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:13.645143 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:13.646343 501 hybrid_clock.cc:648] HybridClock initialized: now 1781831173646320 us; error 60 us; skew 500 ppm
I20260619 01:06:13.646696 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:13 dist-test-slave-5p8w krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:13.649999 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:06:13.651543 501 webserver.cc:460] Webserver started at http://127.0.97.1:38077/ using document root <none> and password file <none>
I20260619 01:06:13.652267 501 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:13.652405 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:13.652719 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:13.656234 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "def25097b2634d029a1f18ca2244f19e"
format_stamp: "Formatted at 2026-06-19 01:06:13 on dist-test-slave-5p8w"
server_key: "729563a02b81c614c0b54f5c95ab6b42"
server_key_iv: "c3e8f5a980fec436dfa2183e03528950"
server_key_version: "encryptionkey@0"
I20260619 01:06:13.657051 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "def25097b2634d029a1f18ca2244f19e"
format_stamp: "Formatted at 2026-06-19 01:06:13 on dist-test-slave-5p8w"
server_key: "729563a02b81c614c0b54f5c95ab6b42"
server_key_iv: "c3e8f5a980fec436dfa2183e03528950"
server_key_version: "encryptionkey@0"
I20260619 01:06:13.662556 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 01:06:13.666168 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:13.667526 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:06:13.667708 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "def25097b2634d029a1f18ca2244f19e"
format_stamp: "Formatted at 2026-06-19 01:06:13 on dist-test-slave-5p8w"
server_key: "729563a02b81c614c0b54f5c95ab6b42"
server_key_iv: "c3e8f5a980fec436dfa2183e03528950"
server_key_version: "encryptionkey@0"
I20260619 01:06:13.667908 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:13.690423 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:13.694808 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:13.695114 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:13.696125 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:13.697552 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:13.697664 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:13.697806 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:13.697876 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:13.734694 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36895
I20260619 01:06:13.734742 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36895 every 8 connection(s)
I20260619 01:06:13.736070 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:06:13.742960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 501
I20260619 01:06:13.743212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:06:13.743818 388 external_mini_cluster.cc:1428] Setting key 58bf498a01abec3eea9f6576bf814168
Jun 19 01:06:13 dist-test-slave-5p8w krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:13.761775 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:13.738781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52711 (local address 127.0.97.62:36863)
0619 01:06:13.739205 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:13.739210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:13.740313 (+ 1103us) server_negotiation.cc:407] Connection header received
0619 01:06:13.742691 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:13.742697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:13.742797 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:13.742915 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:13.744879 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:13.745858 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:13.749256 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:13.749757 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:13.749866 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:13.754342 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:13.754352 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:13.754361 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:13.754410 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:13.758159 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:13.759033 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:13.759042 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:13.759047 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:13.759220 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:13.759725 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:13.759732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:13.759738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:13.760129 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:13.760279 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:13.761043 (+ 764us) server_negotiation.cc:299] Negotiation successful
0619 01:06:13.761343 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:06:13.762760 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:13.739437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36863 (local address 127.0.97.1:52711)
0619 01:06:13.740135 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:13.740181 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:13.742342 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:13.743252 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:13.743263 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:13.743889 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:13.744691 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:13.744723 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:13.746107 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:13.746113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:13.749054 (+ 2941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:13.749078 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:13.750040 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:13.750051 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:13.750427 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:13.751314 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:13.751361 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:13.754061 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:13.758372 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:13.758381 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:13.758398 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:13.758883 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:13.759388 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:13.759392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:13.759395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:13.759602 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:13.760264 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:13.760270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:13.760639 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 01:06:13.760889 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 01:06:13.761458 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":258,"threads_started":1}
I20260619 01:06:13.764691 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:06:13.765556 629 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:13.766345 629 heartbeater.cc:511] Master 127.0.97.62:36863 requested a full tablet report, sending...
I20260619 01:06:13.768810 440 ts_manager.cc:194] Registered new tserver with Master: def25097b2634d029a1f18ca2244f19e (127.0.97.1:36895)
I20260619 01:06:13.771092 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:06:13.788559 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36863
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:14.147863 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:14.148198 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:14.148298 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:14.156322 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:14.156456 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:14.156574 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:06:14.167250 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:14.168648 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:14.170186 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:14.186439 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:14.187091 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:14.187376 635 server_base.cc:1034] running on GCE node
W20260619 01:06:14.187529 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:14.188334 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:14.189568 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:14.190845 635 hybrid_clock.cc:648] HybridClock initialized: now 1781831174190751 us; error 94 us; skew 500 ppm
I20260619 01:06:14.191231 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:14 dist-test-slave-5p8w krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:14.194660 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:06:14.196460 635 webserver.cc:460] Webserver started at http://127.0.97.2:36967/ using document root <none> and password file <none>
I20260619 01:06:14.197172 635 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:14.197275 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:14.197636 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:14.200712 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "64abd7823d3a4265b6bc164cb8a3b197"
format_stamp: "Formatted at 2026-06-19 01:06:14 on dist-test-slave-5p8w"
server_key: "fcf0114f2cb3c92cb01d9f7f06d2457e"
server_key_iv: "2b07d22369e044ea27a99a5ccee926b2"
server_key_version: "encryptionkey@0"
I20260619 01:06:14.201581 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "64abd7823d3a4265b6bc164cb8a3b197"
format_stamp: "Formatted at 2026-06-19 01:06:14 on dist-test-slave-5p8w"
server_key: "fcf0114f2cb3c92cb01d9f7f06d2457e"
server_key_iv: "2b07d22369e044ea27a99a5ccee926b2"
server_key_version: "encryptionkey@0"
I20260619 01:06:14.207679 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:06:14.211634 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:14.213436 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:14.213610 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "64abd7823d3a4265b6bc164cb8a3b197"
format_stamp: "Formatted at 2026-06-19 01:06:14 on dist-test-slave-5p8w"
server_key: "fcf0114f2cb3c92cb01d9f7f06d2457e"
server_key_iv: "2b07d22369e044ea27a99a5ccee926b2"
server_key_version: "encryptionkey@0"
I20260619 01:06:14.213840 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:14.230770 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:14.235874 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:14.236254 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:14.237550 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:14.239454 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:14.239537 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260619 01:06:14.239686 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:14.239740 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:14.278445 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40029
I20260619 01:06:14.278496 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40029 every 8 connection(s)
I20260619 01:06:14.279897 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:06:14.288406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 635
I20260619 01:06:14.288653 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:06:14.289103 388 external_mini_cluster.cc:1428] Setting key d6da3b650699e3069a37b5552cf86f54
Jun 19 01:06:14 dist-test-slave-5p8w krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:14.303248 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:14.282751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40423 (local address 127.0.97.62:36863)
0619 01:06:14.283386 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:14.283391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:14.284403 (+ 1012us) server_negotiation.cc:407] Connection header received
0619 01:06:14.286435 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:14.286440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:14.286545 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:14.286671 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:14.288496 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.289443 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.291174 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.291891 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.292053 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:14.296488 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:14.296494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:14.296498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:14.296535 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:14.299357 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:14.300513 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:14.300518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:14.300532 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:14.300675 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:14.301231 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:14.301234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:14.301236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:14.301596 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:14.301718 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:14.302775 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0619 01:06:14.302962 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":245,"threads_started":1}
I20260619 01:06:14.305092 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:14.283382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36863 (local address 127.0.97.2:40423)
0619 01:06:14.284210 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:14.284252 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:14.286115 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:14.286945 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:14.286956 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:14.287613 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:14.288268 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:14.288298 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.289633 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.289638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:14.291009 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:14.291021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.292162 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.292171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:14.292443 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:14.293226 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:14.293250 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:14.296180 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:14.299554 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:14.299567 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:14.299591 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:14.300345 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:14.300804 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:14.300812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:14.300818 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:14.301117 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:14.301739 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:14.301752 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:14.302302 (+ 550us) client_negotiation.cc:769] Sending connection context
0619 01:06:14.302687 (+ 385us) client_negotiation.cc:240] Negotiation successful
0619 01:06:14.303322 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":355,"threads_started":1}
I20260619 01:06:14.306849 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36863
I20260619 01:06:14.307185 763 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:06:14.308007 763 heartbeater.cc:511] Master 127.0.97.62:36863 requested a full tablet report, sending...
I20260619 01:06:14.309765 440 ts_manager.cc:194] Registered new tserver with Master: 64abd7823d3a4265b6bc164cb8a3b197 (127.0.97.2:40029)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 01:06:14.310683 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:06:14.333161 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36863
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:14.688638 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:14.688995 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:14.689138 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:14.697091 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:14.697288 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:14.697485 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:06:14.707720 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:14.709146 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:14.710628 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:14.725445 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:14.725229 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:14.725229 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:14.726267 769 server_base.cc:1034] running on GCE node
I20260619 01:06:14.726886 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:14.727876 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:14.729112 769 hybrid_clock.cc:648] HybridClock initialized: now 1781831174729064 us; error 51 us; skew 500 ppm
I20260619 01:06:14.729521 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:14 dist-test-slave-5p8w krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:14.736896 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:06:14.738623 769 webserver.cc:460] Webserver started at http://127.0.97.3:39957/ using document root <none> and password file <none>
I20260619 01:06:14.739334 769 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:14.739431 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:14.739784 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:14.743245 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "c1db061b30a341c6bcdf5e7c918c411e"
format_stamp: "Formatted at 2026-06-19 01:06:14 on dist-test-slave-5p8w"
server_key: "50f4bb40726499f07754cfef0bf07c08"
server_key_iv: "4d8e6ce21b07c6adef15abaee1a63670"
server_key_version: "encryptionkey@0"
I20260619 01:06:14.744084 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c1db061b30a341c6bcdf5e7c918c411e"
format_stamp: "Formatted at 2026-06-19 01:06:14 on dist-test-slave-5p8w"
server_key: "50f4bb40726499f07754cfef0bf07c08"
server_key_iv: "4d8e6ce21b07c6adef15abaee1a63670"
server_key_version: "encryptionkey@0"
I20260619 01:06:14.749763 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 01:06:14.753515 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:14.754978 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:06:14.755178 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "c1db061b30a341c6bcdf5e7c918c411e"
format_stamp: "Formatted at 2026-06-19 01:06:14 on dist-test-slave-5p8w"
server_key: "50f4bb40726499f07754cfef0bf07c08"
server_key_iv: "4d8e6ce21b07c6adef15abaee1a63670"
server_key_version: "encryptionkey@0"
I20260619 01:06:14.755409 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:14.775322 629 heartbeater.cc:503] Master 127.0.97.62:36863 was elected leader, sending a full tablet report...
I20260619 01:06:14.778131 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:14.783324 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:14.783622 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:14.784617 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:14.786005 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:14.786110 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:14.786253 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:14.786326 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:14.824028 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45941
I20260619 01:06:14.824119 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45941 every 8 connection(s)
I20260619 01:06:14.825452 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:06:14.832314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 769
I20260619 01:06:14.832513 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:06:14.832957 388 external_mini_cluster.cc:1428] Setting key 7ade916a584eb3da5d7ee5c521da5622
Jun 19 01:06:14 dist-test-slave-5p8w krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:14.847291 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:14.828029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44437 (local address 127.0.97.62:36863)
0619 01:06:14.828523 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:14.828529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:14.829438 (+ 909us) server_negotiation.cc:407] Connection header received
0619 01:06:14.831814 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:14.831820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:14.831921 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:14.832058 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:14.834381 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.835265 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.837549 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.837945 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.838050 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:14.841680 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:14.841685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:14.841689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:14.841723 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:14.844151 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:14.844926 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:14.844930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:14.844932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:14.845068 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:14.845595 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:14.845599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:14.845601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:14.845864 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:14.845985 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:14.846746 (+ 761us) server_negotiation.cc:299] Negotiation successful
0619 01:06:14.846928 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":266,"threads_started":1}
I20260619 01:06:14.848495 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:14.828527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36863 (local address 127.0.97.3:44437)
0619 01:06:14.829218 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:14.829267 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:14.831472 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:14.832407 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:14.832421 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:14.833217 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:14.834137 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:14.834165 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.835422 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.835426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:14.837389 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:14.837404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.838165 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.838169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:14.838370 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:14.839087 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:14.839114 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:14.841403 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:14.844310 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:14.844318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:14.844335 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:14.844802 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:14.845236 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:14.845240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:14.845242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:14.845488 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:14.845967 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:14.845973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:14.846398 (+ 425us) client_negotiation.cc:769] Sending connection context
0619 01:06:14.846638 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 01:06:14.847058 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":263,"threads_started":1}
I20260619 01:06:14.850013 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36863
I20260619 01:06:14.850355 897 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:14.851153 897 heartbeater.cc:511] Master 127.0.97.62:36863 requested a full tablet report, sending...
I20260619 01:06:14.852741 439 ts_manager.cc:194] Registered new tserver with Master: c1db061b30a341c6bcdf5e7c918c411e (127.0.97.3:45941)
I20260619 01:06:14.853711 439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44437
I20260619 01:06:14.862435 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 01:06:14 dist-test-slave-5p8w krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 01:06:14 dist-test-slave-5p8w krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:06:14.910140 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:14.893973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.0.97.1:36895)
0619 01:06:14.894648 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:14.894656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:14.894688 (+ 32us) server_negotiation.cc:407] Connection header received
0619 01:06:14.894869 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:14.894876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:14.895110 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:14.895260 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:14.896657 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.897765 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.899045 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.899661 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.899766 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:14.903957 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:14.903973 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:14.903989 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:14.904032 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:14.906916 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:14.907641 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:14.907649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:14.907651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:14.907776 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:14.908568 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:14.908572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:14.908574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:14.908924 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:14.909166 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:14.909479 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 01:06:14.909718 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":364,"threads_started":1}
W20260619 01:06:14.911499 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49044
Jun 19 01:06:14 dist-test-slave-5p8w krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 01:06:14 dist-test-slave-5p8w krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:06:14.958125 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:14.943966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.0.97.1:36895)
0619 01:06:14.944226 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:14.944233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:14.944536 (+ 303us) server_negotiation.cc:407] Connection header received
0619 01:06:14.944755 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:14.944760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:14.944858 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:14.944968 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:14.946407 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.947176 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.948398 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.948819 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.948956 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:14.953090 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:14.953095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:14.953099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:14.953138 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:14.955797 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:14.956511 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:14.956515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:14.956517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:14.956644 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:14.956972 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:14.956975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:14.956977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:14.957218 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:14.957452 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:14.957731 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 01:06:14.957897 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 01:06:14.959729 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 501
I20260619 01:06:14.974148 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:14.960712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.0.97.1:36895)
0619 01:06:14.960935 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:14.960944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:14.961220 (+ 276us) server_negotiation.cc:407] Connection header received
0619 01:06:14.961394 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:14.961398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:14.961453 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:14.961529 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:14.963087 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.963779 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.965078 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:14.965587 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:14.965685 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:14.968635 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:14.968639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:14.968643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:14.968673 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:14.971350 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:14.972315 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:14.972319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:14.972321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:14.972461 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:14.972834 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:14.972837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:14.972839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:14.973092 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:14.973267 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:14.973598 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 01:06:14.973769 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:06:14.974906 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:15.062295 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 501
I20260619 01:06:15.079739 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 635
Jun 19 01:06:15 dist-test-slave-5p8w krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:06:15.098065 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:15.080708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.0.97.2:40029)
0619 01:06:15.081338 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:15.081367 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:15.081408 (+ 41us) server_negotiation.cc:407] Connection header received
0619 01:06:15.081469 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:15.081476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:15.081716 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:15.081854 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:15.083074 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:15.084128 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:15.085536 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:15.086125 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:15.086233 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:15.090861 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:15.090883 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:15.090906 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:15.090959 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:15.094963 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:15.095786 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:15.095797 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:15.095801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:15.095953 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:15.096389 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:15.096395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:15.096399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:15.096853 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:15.097164 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:15.097518 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 01:06:15.097779 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":334,"threads_started":1}
I20260619 01:06:15.099385 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:15.186707 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 635
I20260619 01:06:15.212075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 769
Jun 19 01:06:15 dist-test-slave-5p8w krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:06:15.230356 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:15.213152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.0.97.3:45941)
0619 01:06:15.213901 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:15.213909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:15.213939 (+ 30us) server_negotiation.cc:407] Connection header received
0619 01:06:15.214001 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:15.214007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:15.214246 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:15.214435 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:15.215815 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:15.216913 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:15.218303 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:15.218919 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:15.219074 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:15.223571 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:15.223588 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:15.223605 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:15.223650 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:15.226763 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:15.227727 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:15.227737 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:15.227739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:15.227922 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:15.228392 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:15.228400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:15.228403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:15.228817 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:15.229197 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:15.229627 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 01:06:15.229955 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":386,"threads_started":1}
I20260619 01:06:15.231768 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:15.317153 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 769
I20260619 01:06:15.334534 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 408
I20260619 01:06:15.335857 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:15.447306 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 408
2026-06-19T01:06:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4123 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:06:15 dist-test-slave-5p8w krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:06:15 dist-test-slave-5p8w krb5kdc[919](info): set up 2 sockets
Jun 19 01:06:15 dist-test-slave-5p8w krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260619 01:06:16.553412 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:06:16 dist-test-slave-5p8w krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:06:16.745465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35519
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:17.095783 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:17.096179 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:17.096364 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:17.103965 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:17.104161 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:17.104303 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:17.104411 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:17.104509 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:06:17.114871 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:17.116400 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:17.117866 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:17.130528 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:17.130657 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:17.131083 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:17.131731 932 server_base.cc:1034] running on GCE node
I20260619 01:06:17.132673 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:17.134403 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:17.135646 932 hybrid_clock.cc:648] HybridClock initialized: now 1781831177135539 us; error 105 us; skew 500 ppm
I20260619 01:06:17.136096 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:17 dist-test-slave-5p8w krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:17.140949 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:06:17.142993 932 webserver.cc:460] Webserver started at http://127.0.97.62:41319/ using document root <none> and password file <none>
I20260619 01:06:17.143784 932 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:17.143932 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:17.144277 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:17.147588 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "21ba48e36dfa410e8f6709fbbf3f578e"
format_stamp: "Formatted at 2026-06-19 01:06:17 on dist-test-slave-5p8w"
server_key: "7d3e16a6cb73f81bd6157709354a06c1"
server_key_iv: "4869eac393dc80ad61269e946a96faa2"
server_key_version: "encryptionkey@0"
I20260619 01:06:17.148429 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "21ba48e36dfa410e8f6709fbbf3f578e"
format_stamp: "Formatted at 2026-06-19 01:06:17 on dist-test-slave-5p8w"
server_key: "7d3e16a6cb73f81bd6157709354a06c1"
server_key_iv: "4869eac393dc80ad61269e946a96faa2"
server_key_version: "encryptionkey@0"
I20260619 01:06:17.154341 932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 01:06:17.158211 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:17.159847 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:06:17.160094 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "21ba48e36dfa410e8f6709fbbf3f578e"
format_stamp: "Formatted at 2026-06-19 01:06:17 on dist-test-slave-5p8w"
server_key: "7d3e16a6cb73f81bd6157709354a06c1"
server_key_iv: "4869eac393dc80ad61269e946a96faa2"
server_key_version: "encryptionkey@0"
I20260619 01:06:17.160352 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:17.180748 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:17.185732 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:17.186060 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:17.206700 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35519
I20260619 01:06:17.206723 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35519 every 8 connection(s)
I20260619 01:06:17.208195 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:17.212159 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:17.214224 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 932
I20260619 01:06:17.214460 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:17.215263 388 external_mini_cluster.cc:1428] Setting key 57143c8ce159d231fc3f5d231f602ceb
I20260619 01:06:17.225729 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e: Bootstrap starting.
I20260619 01:06:17.229954 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e: Neither blocks nor log segments found. Creating new log.
Jun 19 01:06:17 dist-test-slave-5p8w krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:17.231192 999 log.cc:824] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:17.235153 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e: No bootstrap required, opened a new log
I20260619 01:06:17.242040 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:17.217580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.0.97.62:35519)
0619 01:06:17.218445 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:17.218467 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:17.218515 (+ 48us) server_negotiation.cc:407] Connection header received
0619 01:06:17.220271 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:17.220297 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:17.220764 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:17.221129 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:17.223053 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:17.224437 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:17.226293 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:17.227030 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:17.227181 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:17.232189 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:17.232199 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:17.232217 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:17.232254 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:17.236358 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:17.237248 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:17.237263 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:17.237278 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:17.237579 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:17.238215 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:17.238222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:17.238225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:17.239055 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:17.239389 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:17.239922 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 01:06:17.240431 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":395,"threads_started":1}
I20260619 01:06:17.245635 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ba48e36dfa410e8f6709fbbf3f578e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } }
I20260619 01:06:17.245993 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:17.246066 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ba48e36dfa410e8f6709fbbf3f578e, State: Initialized, Role: FOLLOWER
I20260619 01:06:17.246560 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ba48e36dfa410e8f6709fbbf3f578e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } }
I20260619 01:06:17.246757 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:17.246843 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:17.246968 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:17.249079 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ba48e36dfa410e8f6709fbbf3f578e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } }
I20260619 01:06:17.249742 999 leader_election.cc:302] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21ba48e36dfa410e8f6709fbbf3f578e; no voters:
I20260619 01:06:17.250414 999 leader_election.cc:288] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:17.250587 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:17.251257 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [term 1 LEADER]: Becoming Leader. State: Replica: 21ba48e36dfa410e8f6709fbbf3f578e, State: Running, Role: LEADER
I20260619 01:06:17.251858 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ba48e36dfa410e8f6709fbbf3f578e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } }
I20260619 01:06:17.252631 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:17.254927 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21ba48e36dfa410e8f6709fbbf3f578e. Latest consensus state: current_term: 1 leader_uuid: "21ba48e36dfa410e8f6709fbbf3f578e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ba48e36dfa410e8f6709fbbf3f578e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } } }
I20260619 01:06:17.255431 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:17.255031 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21ba48e36dfa410e8f6709fbbf3f578e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21ba48e36dfa410e8f6709fbbf3f578e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35519 } } }
I20260619 01:06:17.255705 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:17.257757 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:17.262701 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:17.272487 1013 catalog_manager.cc:1347] Generated new cluster ID: fd667a0d669146088013bbd7940ed45a
I20260619 01:06:17.272627 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:17.302067 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:17.303542 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:17.325553 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21ba48e36dfa410e8f6709fbbf3f578e: Generated new TSK 0
I20260619 01:06:17.326500 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:06:17.381669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:17.724171 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:17.724555 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:17.724732 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:17.732509 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:17.732722 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:17.732901 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:06:17.743736 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:17.745167 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:17.746577 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:17.761528 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:17.761613 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:17.761562 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:17.762398 1025 server_base.cc:1034] running on GCE node
I20260619 01:06:17.763063 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:17.764081 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:17.765331 1025 hybrid_clock.cc:648] HybridClock initialized: now 1781831177765220 us; error 63 us; skew 500 ppm
I20260619 01:06:17.765697 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:17 dist-test-slave-5p8w krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:17.769083 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:06:17.770653 1025 webserver.cc:460] Webserver started at http://127.0.97.1:40749/ using document root <none> and password file <none>
I20260619 01:06:17.771356 1025 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:17.771457 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:17.771737 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:17.774870 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "aa7591590d4844c58bda331e8b55444b"
format_stamp: "Formatted at 2026-06-19 01:06:17 on dist-test-slave-5p8w"
server_key: "496546fca921212958a346b8025b8972"
server_key_iv: "40875d76d768632eed31be6a479dd313"
server_key_version: "encryptionkey@0"
I20260619 01:06:17.775647 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "aa7591590d4844c58bda331e8b55444b"
format_stamp: "Formatted at 2026-06-19 01:06:17 on dist-test-slave-5p8w"
server_key: "496546fca921212958a346b8025b8972"
server_key_iv: "40875d76d768632eed31be6a479dd313"
server_key_version: "encryptionkey@0"
I20260619 01:06:17.781244 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 01:06:17.785017 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:17.786538 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:17.786736 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "aa7591590d4844c58bda331e8b55444b"
format_stamp: "Formatted at 2026-06-19 01:06:17 on dist-test-slave-5p8w"
server_key: "496546fca921212958a346b8025b8972"
server_key_iv: "40875d76d768632eed31be6a479dd313"
server_key_version: "encryptionkey@0"
I20260619 01:06:17.786960 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:17.811306 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:17.816250 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:17.816588 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:17.817667 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:17.819136 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:17.819245 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:17.819398 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:17.819470 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:17.853928 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35393
I20260619 01:06:17.853984 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35393 every 8 connection(s)
I20260619 01:06:17.855305 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:06:17.861207 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 1025
I20260619 01:06:17.861462 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:06:17.862146 388 external_mini_cluster.cc:1428] Setting key 634f6cd6830b0b0372896c922871a358
Jun 19 01:06:17 dist-test-slave-5p8w krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:17.876333 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:17.858036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49871 (local address 127.0.97.62:35519)
0619 01:06:17.858439 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:17.858443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:17.859530 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 01:06:17.861502 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:17.861508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:17.861623 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:17.861749 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:17.863392 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:17.864178 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:17.866047 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:17.866440 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:17.866585 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:17.870424 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:17.870429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:17.870433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:17.870467 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:17.873032 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:17.873931 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:17.873937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:17.873939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:17.874066 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:17.874524 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:17.874533 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:17.874535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:17.874796 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:17.874952 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:17.875717 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 01:06:17.875958 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":201,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:06:17.877556 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:17.858664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35519 (local address 127.0.97.1:49871)
0619 01:06:17.859368 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:17.859407 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:17.861141 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:17.862007 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:17.862016 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:17.862615 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:17.863220 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:17.863238 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:17.864381 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:17.864385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:17.865891 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:17.865904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:17.866678 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:17.866686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:17.866936 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:17.867662 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:17.867685 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:17.870172 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:17.873208 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:17.873217 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:17.873238 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:17.873784 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:17.874187 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:17.874192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:17.874196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:17.874398 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:17.874927 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:17.874936 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:17.875330 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 01:06:17.875612 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 01:06:17.876191 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":277,"threads_started":1}
I20260619 01:06:17.879210 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35519
I20260619 01:06:17.879552 1153 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:06:17.880436 1153 heartbeater.cc:511] Master 127.0.97.62:35519 requested a full tablet report, sending...
I20260619 01:06:17.883230 964 ts_manager.cc:194] Registered new tserver with Master: aa7591590d4844c58bda331e8b55444b (127.0.97.1:35393)
I20260619 01:06:17.885345 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:06:17.904484 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:18.236093 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:18.236469 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:18.236649 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:18.244073 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:18.244244 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:18.244428 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:06:18.254649 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:18.256050 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:18.257555 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:18.271754 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:18.271675 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:18.271792 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:18.273168 1159 server_base.cc:1034] running on GCE node
I20260619 01:06:18.273846 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:18.274904 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:18.276130 1159 hybrid_clock.cc:648] HybridClock initialized: now 1781831178276066 us; error 70 us; skew 500 ppm
I20260619 01:06:18.276465 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:18 dist-test-slave-5p8w krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:18.279917 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:06:18.281502 1159 webserver.cc:460] Webserver started at http://127.0.97.2:41903/ using document root <none> and password file <none>
I20260619 01:06:18.282243 1159 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:18.282407 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:18.282737 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:18.286002 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "f7c3b9ce4e28457a85732e48b3b7bde0"
format_stamp: "Formatted at 2026-06-19 01:06:18 on dist-test-slave-5p8w"
server_key: "2286b19a13ff017a7576ce5803f86fd9"
server_key_iv: "a0cbb96a732753237487de8b0b22f76b"
server_key_version: "encryptionkey@0"
I20260619 01:06:18.286827 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f7c3b9ce4e28457a85732e48b3b7bde0"
format_stamp: "Formatted at 2026-06-19 01:06:18 on dist-test-slave-5p8w"
server_key: "2286b19a13ff017a7576ce5803f86fd9"
server_key_iv: "a0cbb96a732753237487de8b0b22f76b"
server_key_version: "encryptionkey@0"
I20260619 01:06:18.292621 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 01:06:18.296442 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:18.297930 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:18.298112 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "f7c3b9ce4e28457a85732e48b3b7bde0"
format_stamp: "Formatted at 2026-06-19 01:06:18 on dist-test-slave-5p8w"
server_key: "2286b19a13ff017a7576ce5803f86fd9"
server_key_iv: "a0cbb96a732753237487de8b0b22f76b"
server_key_version: "encryptionkey@0"
I20260619 01:06:18.298334 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:18.313027 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:18.317888 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:18.318207 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:18.319214 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:18.320634 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:18.320745 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:18.320902 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:18.320981 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:18.355285 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41783
I20260619 01:06:18.355321 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41783 every 8 connection(s)
I20260619 01:06:18.356906 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:06:18.362737 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 1159
I20260619 01:06:18.362949 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:06:18.363454 388 external_mini_cluster.cc:1428] Setting key 08ac9bb039d52b505f5ce47229d245f3
Jun 19 01:06:18 dist-test-slave-5p8w krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:18.381248 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:18.359765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47663 (local address 127.0.97.62:35519)
0619 01:06:18.359973 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:18.359978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:18.361248 (+ 1270us) server_negotiation.cc:407] Connection header received
0619 01:06:18.363754 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:18.363761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:18.363857 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:18.364019 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:18.365948 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.366635 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.369356 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.369834 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.369940 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:18.373704 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:18.373713 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:18.373722 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:18.373766 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:18.377512 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:18.378529 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:18.378534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:18.378536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:18.378689 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:18.379328 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:18.379333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:18.379335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:18.379620 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:18.379736 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:18.380799 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0619 01:06:18.380994 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:06:18.382769 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:18.360292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35519 (local address 127.0.97.2:47663)
0619 01:06:18.361059 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:18.361100 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:18.363344 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:18.364263 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:18.364273 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:18.364957 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:18.365770 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:18.365796 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.366800 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.366808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:18.369122 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:18.369141 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.370074 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.370083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:18.370441 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:18.371207 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:18.371229 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:18.373460 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:18.377678 (+ 4218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:18.377685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:18.377700 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:18.378350 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:18.378822 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:18.378838 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:18.378845 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:18.379141 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:18.379765 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:18.379776 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:18.380312 (+ 536us) client_negotiation.cc:769] Sending connection context
0619 01:06:18.380655 (+ 343us) client_negotiation.cc:240] Negotiation successful
0619 01:06:18.381286 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":306,"threads_started":1}
I20260619 01:06:18.384353 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35519
I20260619 01:06:18.384739 1287 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 01:06:18.385671 1287 heartbeater.cc:511] Master 127.0.97.62:35519 requested a full tablet report, sending...
I20260619 01:06:18.387467 964 ts_manager.cc:194] Registered new tserver with Master: f7c3b9ce4e28457a85732e48b3b7bde0 (127.0.97.2:41783)
I20260619 01:06:18.388752 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:06:18.408908 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:18.742520 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:18.742893 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:18.743079 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:18.750563 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:18.750756 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:18.750957 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:06:18.761327 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35519
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:18.762744 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:18.764120 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:18.779422 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:18.779428 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:18.779527 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:18.780237 1292 server_base.cc:1034] running on GCE node
I20260619 01:06:18.780926 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:18.782083 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:18.783309 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781831178783236 us; error 70 us; skew 500 ppm
I20260619 01:06:18.783665 1292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:18 dist-test-slave-5p8w krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:18.787285 1292 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:06:18.788884 1292 webserver.cc:460] Webserver started at http://127.0.97.3:33543/ using document root <none> and password file <none>
I20260619 01:06:18.789706 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:18.789851 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:18.790189 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:18.793279 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "8a16943223c04eeea77b998e76e0a92f"
format_stamp: "Formatted at 2026-06-19 01:06:18 on dist-test-slave-5p8w"
server_key: "fcf1de41f20f77dd531e466d86b5ca2a"
server_key_iv: "80f2978dd163ce698e612634fdd39c00"
server_key_version: "encryptionkey@0"
I20260619 01:06:18.794243 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8a16943223c04eeea77b998e76e0a92f"
format_stamp: "Formatted at 2026-06-19 01:06:18 on dist-test-slave-5p8w"
server_key: "fcf1de41f20f77dd531e466d86b5ca2a"
server_key_iv: "80f2978dd163ce698e612634fdd39c00"
server_key_version: "encryptionkey@0"
I20260619 01:06:18.800274 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 01:06:18.804257 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:18.805888 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:18.806098 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "8a16943223c04eeea77b998e76e0a92f"
format_stamp: "Formatted at 2026-06-19 01:06:18 on dist-test-slave-5p8w"
server_key: "fcf1de41f20f77dd531e466d86b5ca2a"
server_key_iv: "80f2978dd163ce698e612634fdd39c00"
server_key_version: "encryptionkey@0"
I20260619 01:06:18.806319 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:18.825724 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:18.830260 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:18.830585 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:18.831653 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:18.833168 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:18.833244 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:18.833429 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:18.833487 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:18.872522 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40365
I20260619 01:06:18.872556 1419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40365 every 8 connection(s)
I20260619 01:06:18.874027 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:06:18.877559 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 1292
I20260619 01:06:18.877751 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:06:18.878185 388 external_mini_cluster.cc:1428] Setting key d6dbf46bd8255df779346c47ac9fe000
I20260619 01:06:18.889580 1153 heartbeater.cc:503] Master 127.0.97.62:35519 was elected leader, sending a full tablet report...
Jun 19 01:06:18 dist-test-slave-5p8w krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:18.900504 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:18.876786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57421 (local address 127.0.97.62:35519)
0619 01:06:18.876968 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:18.876973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:18.882225 (+ 5252us) server_negotiation.cc:407] Connection header received
0619 01:06:18.884225 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:18.884229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:18.884304 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:18.884394 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:18.886352 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.887102 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.888784 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.889169 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.889352 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:18.893921 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:18.893927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:18.893930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:18.893966 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:18.897128 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:18.897993 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:18.898003 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:18.898006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:18.898133 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:18.898589 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:18.898598 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:18.898600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:18.898883 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:18.899043 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:18.899895 (+ 852us) server_negotiation.cc:299] Negotiation successful
0619 01:06:18.900130 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:06:18.901659 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:18.877407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35519 (local address 127.0.97.3:57421)
0619 01:06:18.882023 (+ 4616us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:18.882077 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:18.883893 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:18.884713 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:18.884724 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:18.885517 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:18.886171 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:18.886190 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.887251 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.887255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:18.888615 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:18.888628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.889513 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.889519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:18.889840 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:18.890697 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:18.890734 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:18.893667 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:18.897338 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:18.897349 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:18.897370 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:18.897848 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:18.898251 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:18.898257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:18.898260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:18.898464 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:18.899032 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:18.899041 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:18.899503 (+ 462us) client_negotiation.cc:769] Sending connection context
0619 01:06:18.899788 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 01:06:18.900383 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4285,"thread_start_us":4055,"threads_started":1}
I20260619 01:06:18.903187 1420 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35519
I20260619 01:06:18.903537 1420 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:18.904330 1420 heartbeater.cc:511] Master 127.0.97.62:35519 requested a full tablet report, sending...
I20260619 01:06:18.905973 964 ts_manager.cc:194] Registered new tserver with Master: 8a16943223c04eeea77b998e76e0a92f (127.0.97.3:40365)
I20260619 01:06:18.906710 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:06:18.907160 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57421
I20260619 01:06:18.927635 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:18.913660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.0.97.62:35519)
0619 01:06:18.913839 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:18.913844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:18.914436 (+ 592us) server_negotiation.cc:407] Connection header received
0619 01:06:18.914603 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:18.914607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:18.914674 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:18.914771 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:18.916445 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.917188 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.918672 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.919090 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.919196 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:18.922161 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:18.922167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:18.922172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:18.922213 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:18.925087 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:18.925884 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:18.925889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:18.925892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:18.926031 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:18.926378 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:18.926382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:18.926384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:18.926715 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:18.926901 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:18.927204 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 01:06:18.927394 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:06:18.934872 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:06:18.941939 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:06:18.971411 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:18.955443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.0.97.1:35393)
0619 01:06:18.956912 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:18.956923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:18.956965 (+ 42us) server_negotiation.cc:407] Connection header received
0619 01:06:18.957070 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:18.957083 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:18.957501 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:18.960041 (+ 2540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:18.960351 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.964165 (+ 3814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.968772 (+ 4607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.970046 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.970223 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:18.970487 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:18.970838 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 01:06:18.971030 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1292,"thread_start_us":1123,"threads_started":1}
I20260619 01:06:18.971508 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:18.955132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35393 (local address 127.0.0.1:60036)
0619 01:06:18.956196 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:18.956248 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:18.956467 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:18.957752 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:18.957759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:18.957788 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:18.958222 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:18.958242 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.964375 (+ 6133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.964383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:18.967880 (+ 3497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:18.967901 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.970275 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.970282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:18.970561 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:18.970611 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 01:06:18.970778 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 01:06:18.971004 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":33,"thread_start_us":448,"threads_started":1}
I20260619 01:06:18.971606 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:18.956453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40365 (local address 127.0.0.1:38032)
0619 01:06:18.957006 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:18.957074 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:18.957188 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:18.958909 (+ 1721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:18.958917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:18.958954 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:18.959511 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:18.959526 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.962274 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.962282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:18.968186 (+ 5904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:18.968196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.970632 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.970638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:18.970850 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:18.970884 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 01:06:18.970975 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 01:06:18.971114 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":5,"thread_start_us":828,"threads_started":1}
I20260619 01:06:18.972612 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:18.957045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.0.97.3:40365)
0619 01:06:18.957876 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:18.957889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:18.957930 (+ 41us) server_negotiation.cc:407] Connection header received
0619 01:06:18.958043 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:18.958053 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:18.958331 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:18.958580 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:18.959791 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.962044 (+ 2253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.968415 (+ 6371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.970082 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.970214 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:18.970485 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:18.971859 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0619 01:06:18.972094 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":476,"threads_started":1}
I20260619 01:06:18.975762 1088 tablet_service.cc:1467] Processing CreateTablet for tablet d8e0878baf0c4d92816c702191c7c7b1 (DEFAULT_TABLE table=test-table [id=6d874540efc544438524770944ec1dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:18.977128 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e0878baf0c4d92816c702191c7c7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:18.977169 1355 tablet_service.cc:1467] Processing CreateTablet for tablet d8e0878baf0c4d92816c702191c7c7b1 (DEFAULT_TABLE table=test-table [id=6d874540efc544438524770944ec1dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:18.978437 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:18.958033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.0.97.2:41783)
0619 01:06:18.968698 (+ 10665us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:18.968708 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:18.968737 (+ 29us) server_negotiation.cc:407] Connection header received
0619 01:06:18.968841 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:18.968862 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:18.969168 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:18.970101 (+ 933us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:18.970329 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.972483 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.975090 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.977164 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.977372 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:18.977608 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:18.977898 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 01:06:18.978102 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10489,"thread_start_us":10393,"threads_started":1}
I20260619 01:06:18.978459 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:18.957618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41783 (local address 127.0.0.1:41014)
0619 01:06:18.967868 (+ 10250us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:18.967890 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:18.968036 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:18.969451 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:18.969456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:18.969505 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:18.969871 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:18.969882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.972657 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.972662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:18.974904 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:18.974921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:18.977489 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:18.977497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:18.977676 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:18.977710 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 01:06:18.977826 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 01:06:18.978103 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10052,"thread_start_us":10504,"threads_started":1}
I20260619 01:06:18.978921 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e0878baf0c4d92816c702191c7c7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:18.982854 1222 tablet_service.cc:1467] Processing CreateTablet for tablet d8e0878baf0c4d92816c702191c7c7b1 (DEFAULT_TABLE table=test-table [id=6d874540efc544438524770944ec1dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:18.984578 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e0878baf0c4d92816c702191c7c7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:18.993214 1438 tablet_bootstrap.cc:492] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f: Bootstrap starting.
I20260619 01:06:18.995751 1439 tablet_bootstrap.cc:492] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b: Bootstrap starting.
I20260619 01:06:18.998303 1438 tablet_bootstrap.cc:654] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:18.999769 1438 log.cc:824] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:19.000216 1439 tablet_bootstrap.cc:654] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:19.000870 1440 tablet_bootstrap.cc:492] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0: Bootstrap starting.
I20260619 01:06:19.001977 1439 log.cc:824] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:19.004726 1438 tablet_bootstrap.cc:492] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f: No bootstrap required, opened a new log
I20260619 01:06:19.004830 1439 tablet_bootstrap.cc:492] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b: No bootstrap required, opened a new log
I20260619 01:06:19.004993 1440 tablet_bootstrap.cc:654] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:19.005049 1438 ts_tablet_manager.cc:1397] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260619 01:06:19.005136 1439 ts_tablet_manager.cc:1397] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260619 01:06:19.006803 1440 log.cc:824] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:19.010967 1440 tablet_bootstrap.cc:492] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0: No bootstrap required, opened a new log
I20260619 01:06:19.011302 1440 ts_tablet_manager.cc:1397] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260619 01:06:19.014772 1438 raft_consensus.cc:348] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7591590d4844c58bda331e8b55444b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35393 } } peers { permanent_uuid: "8a16943223c04eeea77b998e76e0a92f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40365 } } peers { permanent_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260619 01:06:19.015218 1438 raft_consensus.cc:374] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:19.015316 1438 raft_consensus.cc:729] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a16943223c04eeea77b998e76e0a92f, State: Initialized, Role: FOLLOWER
I20260619 01:06:19.015969 1438 consensus_queue.cc:260] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7591590d4844c58bda331e8b55444b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35393 } } peers { permanent_uuid: "8a16943223c04eeea77b998e76e0a92f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40365 } } peers { permanent_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260619 01:06:19.017197 1420 heartbeater.cc:503] Master 127.0.97.62:35519 was elected leader, sending a full tablet report...
I20260619 01:06:19.017782 1438 ts_tablet_manager.cc:1428] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260619 01:06:19.018283 1439 raft_consensus.cc:348] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7591590d4844c58bda331e8b55444b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35393 } } peers { permanent_uuid: "8a16943223c04eeea77b998e76e0a92f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40365 } } peers { permanent_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260619 01:06:19.018808 1439 raft_consensus.cc:374] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:19.018923 1439 raft_consensus.cc:729] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7591590d4844c58bda331e8b55444b, State: Initialized, Role: FOLLOWER
I20260619 01:06:19.019627 1439 consensus_queue.cc:260] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7591590d4844c58bda331e8b55444b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35393 } } peers { permanent_uuid: "8a16943223c04eeea77b998e76e0a92f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40365 } } peers { permanent_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260619 01:06:19.021692 1439 ts_tablet_manager.cc:1428] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b: Time spent starting tablet: real 0.016s user 0.015s sys 0.002s
I20260619 01:06:19.023264 1440 raft_consensus.cc:348] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7591590d4844c58bda331e8b55444b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35393 } } peers { permanent_uuid: "8a16943223c04eeea77b998e76e0a92f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40365 } } peers { permanent_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260619 01:06:19.023828 1440 raft_consensus.cc:374] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:19.023947 1440 raft_consensus.cc:729] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c3b9ce4e28457a85732e48b3b7bde0, State: Initialized, Role: FOLLOWER
I20260619 01:06:19.024804 1440 consensus_queue.cc:260] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7591590d4844c58bda331e8b55444b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35393 } } peers { permanent_uuid: "8a16943223c04eeea77b998e76e0a92f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40365 } } peers { permanent_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260619 01:06:19.026002 1287 heartbeater.cc:503] Master 127.0.97.62:35519 was elected leader, sending a full tablet report...
I20260619 01:06:19.026921 1440 ts_tablet_manager.cc:1428] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0: Time spent starting tablet: real 0.015s user 0.006s sys 0.010s
W20260619 01:06:19.107856 1154 tablet.cc:2369] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:06:19.108968 1288 tablet.cc:2369] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:06:19.125736 1421 tablet.cc:2369] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:06:19.212630 1445 raft_consensus.cc:482] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:19.212977 1445 raft_consensus.cc:504] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7591590d4844c58bda331e8b55444b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35393 } } peers { permanent_uuid: "8a16943223c04eeea77b998e76e0a92f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40365 } } peers { permanent_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260619 01:06:19.215104 1445 leader_election.cc:288] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a16943223c04eeea77b998e76e0a92f (127.0.97.3:40365), f7c3b9ce4e28457a85732e48b3b7bde0 (127.0.97.2:41783)
I20260619 01:06:19.223310 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:19.215545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40365 (local address 127.0.97.1:45397)
0619 01:06:19.216282 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:19.216308 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:19.216479 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:19.217070 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:19.217075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:19.217101 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:19.217953 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:19.217968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.219399 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.219404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:19.221220 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:19.221235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.222537 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.222542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:19.222711 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:19.222752 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 01:06:19.222869 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 01:06:19.223008 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":111,"thread_start_us":424,"threads_started":1}
I20260619 01:06:19.223397 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:19.215572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45397 (local address 127.0.97.3:40365)
0619 01:06:19.215801 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:19.215807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:19.216624 (+ 817us) server_negotiation.cc:407] Connection header received
0619 01:06:19.216727 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:19.216735 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:19.216815 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:19.216900 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:19.218128 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.219203 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.221445 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.222315 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.222455 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:19.222566 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:19.222952 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 01:06:19.223127 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 01:06:19.224663 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:19.216481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41783 (local address 127.0.97.1:52547)
0619 01:06:19.217261 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:19.217282 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:19.217441 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:19.218068 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:19.218073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:19.218101 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:19.218424 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:19.218435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.219742 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.219748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:19.222097 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:19.222113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.223966 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.223975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:19.224150 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:19.224190 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 01:06:19.224281 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 01:06:19.224394 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":4,"thread_start_us":446,"threads_started":1}
I20260619 01:06:19.224758 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:19.216716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52547 (local address 127.0.97.2:41783)
0619 01:06:19.216928 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:19.216933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:19.217749 (+ 816us) server_negotiation.cc:407] Connection header received
0619 01:06:19.217844 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:19.217849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:19.217937 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:19.218034 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:19.218595 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.219593 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.222303 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.223727 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.223877 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:19.223951 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:19.224368 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 01:06:19.224516 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 01:06:19.224761 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e0878baf0c4d92816c702191c7c7b1" candidate_uuid: "aa7591590d4844c58bda331e8b55444b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a16943223c04eeea77b998e76e0a92f" is_pre_election: true
I20260619 01:06:19.225427 1375 raft_consensus.cc:2454] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa7591590d4844c58bda331e8b55444b in term 0.
I20260619 01:06:19.225932 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e0878baf0c4d92816c702191c7c7b1" candidate_uuid: "aa7591590d4844c58bda331e8b55444b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0" is_pre_election: true
I20260619 01:06:19.226251 1041 leader_election.cc:302] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a16943223c04eeea77b998e76e0a92f, aa7591590d4844c58bda331e8b55444b; no voters:
I20260619 01:06:19.226564 1242 raft_consensus.cc:2454] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa7591590d4844c58bda331e8b55444b in term 0.
I20260619 01:06:19.227195 1445 raft_consensus.cc:2781] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:06:19.227375 1445 raft_consensus.cc:482] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:06:19.227504 1445 raft_consensus.cc:3037] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:19.230142 1445 raft_consensus.cc:504] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7591590d4844c58bda331e8b55444b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35393 } } peers { permanent_uuid: "8a16943223c04eeea77b998e76e0a92f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40365 } } peers { permanent_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260619 01:06:19.230927 1445 leader_election.cc:288] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [CANDIDATE]: Term 1 election: Requested vote from peers 8a16943223c04eeea77b998e76e0a92f (127.0.97.3:40365), f7c3b9ce4e28457a85732e48b3b7bde0 (127.0.97.2:41783)
I20260619 01:06:19.231564 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e0878baf0c4d92816c702191c7c7b1" candidate_uuid: "aa7591590d4844c58bda331e8b55444b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a16943223c04eeea77b998e76e0a92f"
I20260619 01:06:19.231871 1375 raft_consensus.cc:3037] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:19.231927 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e0878baf0c4d92816c702191c7c7b1" candidate_uuid: "aa7591590d4844c58bda331e8b55444b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0"
I20260619 01:06:19.232158 1242 raft_consensus.cc:3037] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:19.235577 1242 raft_consensus.cc:2454] T d8e0878baf0c4d92816c702191c7c7b1 P f7c3b9ce4e28457a85732e48b3b7bde0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa7591590d4844c58bda331e8b55444b in term 1.
I20260619 01:06:19.235564 1375 raft_consensus.cc:2454] T d8e0878baf0c4d92816c702191c7c7b1 P 8a16943223c04eeea77b998e76e0a92f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa7591590d4844c58bda331e8b55444b in term 1.
I20260619 01:06:19.236310 1041 leader_election.cc:302] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa7591590d4844c58bda331e8b55444b, f7c3b9ce4e28457a85732e48b3b7bde0; no voters:
I20260619 01:06:19.236742 1445 raft_consensus.cc:2781] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:19.237257 1445 raft_consensus.cc:686] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [term 1 LEADER]: Becoming Leader. State: Replica: aa7591590d4844c58bda331e8b55444b, State: Running, Role: LEADER
I20260619 01:06:19.238034 1445 consensus_queue.cc:237] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7591590d4844c58bda331e8b55444b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35393 } } peers { permanent_uuid: "8a16943223c04eeea77b998e76e0a92f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40365 } } peers { permanent_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260619 01:06:19.243268 964 catalog_manager.cc:5495] T d8e0878baf0c4d92816c702191c7c7b1 P aa7591590d4844c58bda331e8b55444b reported cstate change: term changed from 0 to 1, leader changed from <none> to aa7591590d4844c58bda331e8b55444b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "aa7591590d4844c58bda331e8b55444b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa7591590d4844c58bda331e8b55444b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a16943223c04eeea77b998e76e0a92f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c3b9ce4e28457a85732e48b3b7bde0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } health_report { overall_health: UNKNOWN } } }
Jun 19 01:06:19 dist-test-slave-5p8w krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:06:19.274070 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:19.257621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.0.97.1:35393)
0619 01:06:19.257831 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:19.257836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:19.258934 (+ 1098us) server_negotiation.cc:407] Connection header received
0619 01:06:19.259073 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:19.259078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:19.259148 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:19.259238 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:19.260808 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.261666 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.263073 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.263495 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.263601 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:19.267978 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:19.267995 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:19.268011 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:19.268055 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:19.271057 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.272037 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.272045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.272047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.272207 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.272583 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.272586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.272588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.272926 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:19.273151 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:19.273436 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 01:06:19.273697 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 01:06:19 dist-test-slave-5p8w krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831179, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 01:06:19 dist-test-slave-5p8w krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831179, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:06:19.323359 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:19.308450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.0.97.1:35393)
0619 01:06:19.308657 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:19.308666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:19.309111 (+ 445us) server_negotiation.cc:407] Connection header received
0619 01:06:19.309365 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:19.309370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:19.309452 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:19.309581 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:19.311113 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.311858 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.313170 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.313702 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.313849 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:19.317881 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:19.317886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:19.317890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:19.317932 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:19.320487 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.321255 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.321259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.321262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.321494 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.321923 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.321928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.321933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.322211 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:19.322510 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:19.322760 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 01:06:19.322999 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 01:06:19 dist-test-slave-5p8w krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 01:06:19 dist-test-slave-5p8w krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:06:19.371394 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:19.354332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.0.97.1:35393)
0619 01:06:19.354554 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:19.354564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:19.355017 (+ 453us) server_negotiation.cc:407] Connection header received
0619 01:06:19.355232 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:19.355242 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:19.355324 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:19.355452 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:19.357064 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.358095 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.360485 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.360918 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.361030 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:19.365955 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:19.365961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:19.365965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:19.366004 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:19.368632 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.369463 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.369477 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.369479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.369600 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.369980 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.369987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.369990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.370265 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:19.370523 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:19.370809 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 01:06:19.371051 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 01:06:19.373123 1068 tablet_service.cc:2908] Rejecting scan request for tablet d8e0878baf0c4d92816c702191c7c7b1: Uninitialized: clean time has not yet been initialized
I20260619 01:06:19.376370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 1025
I20260619 01:06:19.389636 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:19.377333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.0.97.1:35393)
0619 01:06:19.377534 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:19.377544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:19.377847 (+ 303us) server_negotiation.cc:407] Connection header received
0619 01:06:19.377961 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:19.377965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:19.378045 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:19.378136 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:19.379325 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.380144 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.381379 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.381794 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.381892 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:19.384028 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:19.384033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:19.384036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:19.384071 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:19.387219 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.387905 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.387909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.387911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.388037 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.388329 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.388333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.388335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.388581 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:19.388749 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:19.388988 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 01:06:19.389176 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 01:06:19.390291 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:19.482820 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 1025
I20260619 01:06:19.500145 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 1159
Jun 19 01:06:19 dist-test-slave-5p8w krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:06:19.521193 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:19.501096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.0.97.2:41783)
0619 01:06:19.501369 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:19.501373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:19.501396 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:06:19.501500 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:19.501505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:19.501610 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:19.501711 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:19.503147 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.504029 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.505648 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.506063 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.506171 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:19.514520 (+ 8349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:19.514539 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:19.514555 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:19.514604 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:19.518085 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.519098 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.519108 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.519110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.519266 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.519729 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.519732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.519734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.520135 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:19.520367 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:19.520715 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 01:06:19.520925 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 01:06:19.521919 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:19.625063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 1159
I20260619 01:06:19.642297 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 1292
Jun 19 01:06:19 dist-test-slave-5p8w krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:06:19.657943 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:19.643132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.0.97.3:40365)
0619 01:06:19.643330 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:19.643335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:19.643356 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:06:19.643570 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:19.643576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:19.643657 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:19.643766 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:19.645047 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.645853 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.647143 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:19.647548 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:19.647688 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:19.652086 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:19.652103 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:19.652121 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:19.652161 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:19.655096 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.655814 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.655822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.655824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.656051 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:19.656404 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:19.656407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:19.656409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:19.656759 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:19.657000 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:19.657429 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 01:06:19.657624 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:06:19.658618 1415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:19.748260 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 1292
I20260619 01:06:19.765141 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 932
I20260619 01:06:19.766346 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:19.878574 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 932
2026-06-19T01:06:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4431 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:06:19 dist-test-slave-5p8w krb5kdc[1468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:06:19 dist-test-slave-5p8w krb5kdc[1468](info): set up 2 sockets
Jun 19 01:06:19 dist-test-slave-5p8w krb5kdc[1468](info): commencing operation
krb5kdc: starting...
W20260619 01:06:21.001506 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:06:21 dist-test-slave-5p8w krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:06:21.198817 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41439
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:21.543573 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:21.543915 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:21.544083 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:21.551707 1481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:21.551844 1481 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 01:06:21.551884 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:21.552007 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:21.552078 1481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:21.552165 1481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:06:21.562567 1481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41439
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:21.564002 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:21.565491 1481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:21.579147 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:21.579211 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:21.579170 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:21.579900 1481 server_base.cc:1034] running on GCE node
I20260619 01:06:21.580829 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:21.582615 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:21.583921 1481 hybrid_clock.cc:648] HybridClock initialized: now 1781831181583856 us; error 49 us; skew 500 ppm
I20260619 01:06:21.584347 1481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:21 dist-test-slave-5p8w krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:21.587962 1481 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:06:21.589730 1481 webserver.cc:460] Webserver started at http://127.0.97.62:39687/ using document root <none> and password file <none>
I20260619 01:06:21.590478 1481 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:21.590569 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:21.590850 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:21.594115 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "bc83b74adcbe4bc4b7b5edde369e170b"
format_stamp: "Formatted at 2026-06-19 01:06:21 on dist-test-slave-5p8w"
server_key: "1c054758ff81bb6849057c5d0cd70610"
server_key_iv: "8956775cec3f58c1d9da124933875be1"
server_key_version: "encryptionkey@0"
I20260619 01:06:21.594971 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "bc83b74adcbe4bc4b7b5edde369e170b"
format_stamp: "Formatted at 2026-06-19 01:06:21 on dist-test-slave-5p8w"
server_key: "1c054758ff81bb6849057c5d0cd70610"
server_key_iv: "8956775cec3f58c1d9da124933875be1"
server_key_version: "encryptionkey@0"
I20260619 01:06:21.601361 1481 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 01:06:21.605890 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:21.607586 1481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:06:21.607790 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "bc83b74adcbe4bc4b7b5edde369e170b"
format_stamp: "Formatted at 2026-06-19 01:06:21 on dist-test-slave-5p8w"
server_key: "1c054758ff81bb6849057c5d0cd70610"
server_key_iv: "8956775cec3f58c1d9da124933875be1"
server_key_version: "encryptionkey@0"
I20260619 01:06:21.608029 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:21.631114 1481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:21.635939 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:21.636260 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:21.659080 1481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41439
I20260619 01:06:21.659087 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41439 every 8 connection(s)
I20260619 01:06:21.660681 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:21.664781 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:21.670122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 1481
I20260619 01:06:21.670380 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:21.671000 388 external_mini_cluster.cc:1428] Setting key 362f6d72d5ab9142632f567726fd2c3a
I20260619 01:06:21.676533 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b: Bootstrap starting.
I20260619 01:06:21.680717 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:21.682194 1558 log.cc:824] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:21.686190 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b: No bootstrap required, opened a new log
Jun 19 01:06:21 dist-test-slave-5p8w krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:21.694752 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc83b74adcbe4bc4b7b5edde369e170b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41439 } }
I20260619 01:06:21.695185 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:21.695329 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc83b74adcbe4bc4b7b5edde369e170b, State: Initialized, Role: FOLLOWER
I20260619 01:06:21.695966 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc83b74adcbe4bc4b7b5edde369e170b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41439 } }
I20260619 01:06:21.696262 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:21.696415 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:21.696602 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:21.696972 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:21.673221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.0.97.62:41439)
0619 01:06:21.674194 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:21.674217 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:21.674269 (+ 52us) server_negotiation.cc:407] Connection header received
0619 01:06:21.676043 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:21.676070 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:21.676536 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:21.676981 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:21.678953 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:21.680331 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:21.682300 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:21.683432 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:21.683666 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:21.688587 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:21.688597 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:21.688618 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:21.688667 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:21.691961 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:21.692810 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:21.692818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:21.692825 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:21.693044 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:21.693539 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:21.693543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:21.693545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:21.694165 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:21.694513 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:21.694943 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 01:06:21.695662 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":429,"threads_started":1}
I20260619 01:06:21.699139 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc83b74adcbe4bc4b7b5edde369e170b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41439 } }
I20260619 01:06:21.699674 1558 leader_election.cc:302] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc83b74adcbe4bc4b7b5edde369e170b; no voters:
I20260619 01:06:21.700321 1558 leader_election.cc:288] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:21.700444 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:21.701088 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [term 1 LEADER]: Becoming Leader. State: Replica: bc83b74adcbe4bc4b7b5edde369e170b, State: Running, Role: LEADER
I20260619 01:06:21.701692 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc83b74adcbe4bc4b7b5edde369e170b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41439 } }
I20260619 01:06:21.702636 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:21.704700 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc83b74adcbe4bc4b7b5edde369e170b. Latest consensus state: current_term: 1 leader_uuid: "bc83b74adcbe4bc4b7b5edde369e170b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc83b74adcbe4bc4b7b5edde369e170b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41439 } } }
I20260619 01:06:21.705190 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:21.707316 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:21.707266 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc83b74adcbe4bc4b7b5edde369e170b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc83b74adcbe4bc4b7b5edde369e170b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41439 } } }
I20260619 01:06:21.707680 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:21.713186 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:21.727401 1570 catalog_manager.cc:1347] Generated new cluster ID: e410db9bb2fe412cabc5ccbf1a55d338
I20260619 01:06:21.727563 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:21.740753 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:21.742491 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:21.750044 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b: Generated new TSK 0
I20260619 01:06:21.751106 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:06:21.802225 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41439
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:22.175165 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:22.175472 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:22.175570 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:22.183063 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:22.183208 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:22.183295 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:06:22.183383 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:06:22.193758 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:22.195046 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:22.196628 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:22.210988 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:22.211354 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:22.211834 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:22.212101 1584 server_base.cc:1034] running on GCE node
I20260619 01:06:22.213162 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:22.214241 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:22.215495 1584 hybrid_clock.cc:648] HybridClock initialized: now 1781831182215430 us; error 49 us; skew 500 ppm
I20260619 01:06:22.215830 1584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:22 dist-test-slave-5p8w krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:22.219221 1584 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:06:22.220782 1584 webserver.cc:460] Webserver started at http://127.0.97.1:37859/ using document root <none> and password file <none>
I20260619 01:06:22.221567 1584 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:22.221659 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:22.222026 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:22.225059 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "fc7dd1c7602749e3ad4c612bda902028"
format_stamp: "Formatted at 2026-06-19 01:06:22 on dist-test-slave-5p8w"
server_key: "0ffa33dceb4ee9c070c19fd8f1fb3ab8"
server_key_iv: "77942236d84c275a646ffef25b713d0f"
server_key_version: "encryptionkey@0"
I20260619 01:06:22.226023 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fc7dd1c7602749e3ad4c612bda902028"
format_stamp: "Formatted at 2026-06-19 01:06:22 on dist-test-slave-5p8w"
server_key: "0ffa33dceb4ee9c070c19fd8f1fb3ab8"
server_key_iv: "77942236d84c275a646ffef25b713d0f"
server_key_version: "encryptionkey@0"
I20260619 01:06:22.231559 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 01:06:22.235411 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:22.236958 1584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:06:22.237157 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "fc7dd1c7602749e3ad4c612bda902028"
format_stamp: "Formatted at 2026-06-19 01:06:22 on dist-test-slave-5p8w"
server_key: "0ffa33dceb4ee9c070c19fd8f1fb3ab8"
server_key_iv: "77942236d84c275a646ffef25b713d0f"
server_key_version: "encryptionkey@0"
I20260619 01:06:22.237489 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:22.254536 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:22.259213 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:22.259508 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:22.262447 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:22.262563 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:22.262715 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:22.262796 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:06:22 dist-test-slave-5p8w krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:22.288216 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:22.263229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40605 (local address 127.0.97.62:41439)
0619 01:06:22.263808 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:22.263818 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:22.265249 (+ 1431us) server_negotiation.cc:407] Connection header received
0619 01:06:22.267327 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:22.267333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:22.267452 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:22.267573 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:22.269991 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.271299 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.274104 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.274752 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.274936 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:22.280557 (+ 5621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:22.280563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:22.280567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:22.280606 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:22.284346 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:22.285575 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:22.285583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:22.285589 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:22.285767 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:22.286286 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:22.286292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:22.286295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:22.286583 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:22.286755 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:22.287567 (+ 812us) server_negotiation.cc:299] Negotiation successful
0619 01:06:22.287812 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":275,"threads_started":1}
I20260619 01:06:22.289887 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:22.263969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41439 (local address 127.0.97.1:40605)
0619 01:06:22.265039 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:22.265097 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:22.267014 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:22.267864 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:22.267882 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:22.268687 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:22.269673 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:22.269701 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.271506 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.271513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:22.273921 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:22.273936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.275068 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.275079 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:22.275491 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:22.276534 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:22.276565 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:22.280229 (+ 3664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:22.284563 (+ 4334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:22.284575 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:22.284600 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:22.285415 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:22.285922 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:22.285926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:22.285928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:22.286146 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:22.286729 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:22.286738 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:22.287137 (+ 399us) client_negotiation.cc:769] Sending connection context
0619 01:06:22.287441 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 01:06:22.288034 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":445,"threads_started":1}
I20260619 01:06:22.313048 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:22.297573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53047 (local address 127.0.97.62:41439)
0619 01:06:22.297752 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:22.297759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:22.298343 (+ 584us) server_negotiation.cc:407] Connection header received
0619 01:06:22.298550 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:22.298556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:22.298636 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:22.298744 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:22.299737 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.300477 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.302879 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.303651 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.303820 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:22.306852 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:22.306857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:22.306861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:22.306909 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:22.310181 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:22.310915 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:22.310920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:22.310924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:22.311060 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:22.311482 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:22.311486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:22.311488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:22.311812 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:22.312003 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:22.312429 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 01:06:22.312662 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:06:22.313125 1681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:22.297567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41439 (local address 127.0.97.1:53047)
0619 01:06:22.298218 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:22.298234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:22.298419 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:22.298778 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:22.298782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:22.299273 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:22.299575 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:22.299584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.300617 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.300628 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:22.302647 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:22.302662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.303953 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.303963 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:22.304230 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:22.305047 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:22.305081 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:22.306578 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:22.310346 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:22.310352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:22.310356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:22.310773 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:22.311181 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:22.311186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:22.311190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:22.311334 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:22.311951 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:22.311956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:22.312147 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 01:06:22.312302 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 01:06:22.312662 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":385,"threads_started":1}
I20260619 01:06:22.324229 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38513
I20260619 01:06:22.324249 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38513 every 8 connection(s)
I20260619 01:06:22.325726 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:06:22.327971 1720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41439
I20260619 01:06:22.328181 1720 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:22.328845 1720 heartbeater.cc:511] Master 127.0.97.62:41439 requested a full tablet report, sending...
I20260619 01:06:22.329690 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 1584
I20260619 01:06:22.329928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:06:22.330749 388 external_mini_cluster.cc:1428] Setting key 25d019f6c164c3ea5aebb5f2dbd11092
I20260619 01:06:22.331481 1513 ts_manager.cc:194] Registered new tserver with Master: fc7dd1c7602749e3ad4c612bda902028 (127.0.97.1:38513)
I20260619 01:06:22.333199 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40605
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:06:22.372638 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41439
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:22.714821 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:22.715193 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:22.715381 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:22.723130 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:22.723309 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:22.723474 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:06:22.723619 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:06:22.734028 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:22.735462 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:22.736937 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:22.751453 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:22.751649 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:22.751906 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:22.752924 1724 server_base.cc:1034] running on GCE node
I20260619 01:06:22.753643 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:22.754611 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:22.755820 1724 hybrid_clock.cc:648] HybridClock initialized: now 1781831182755729 us; error 67 us; skew 500 ppm
I20260619 01:06:22.756156 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:22 dist-test-slave-5p8w krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:22.759523 1724 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:06:22.761085 1724 webserver.cc:460] Webserver started at http://127.0.97.2:41999/ using document root <none> and password file <none>
I20260619 01:06:22.761880 1724 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:22.761976 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:22.762255 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:22.765281 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "15fa75bc2c5e41ca8c3881ac92ba6352"
format_stamp: "Formatted at 2026-06-19 01:06:22 on dist-test-slave-5p8w"
server_key: "b6acb00e3873aa8f6b4b4a88804eb4bc"
server_key_iv: "6fa9fec3e172a863e7d46cee63de1575"
server_key_version: "encryptionkey@0"
I20260619 01:06:22.766165 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "15fa75bc2c5e41ca8c3881ac92ba6352"
format_stamp: "Formatted at 2026-06-19 01:06:22 on dist-test-slave-5p8w"
server_key: "b6acb00e3873aa8f6b4b4a88804eb4bc"
server_key_iv: "6fa9fec3e172a863e7d46cee63de1575"
server_key_version: "encryptionkey@0"
I20260619 01:06:22.771744 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:06:22.775780 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:22.777501 1724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:22.777712 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "15fa75bc2c5e41ca8c3881ac92ba6352"
format_stamp: "Formatted at 2026-06-19 01:06:22 on dist-test-slave-5p8w"
server_key: "b6acb00e3873aa8f6b4b4a88804eb4bc"
server_key_iv: "6fa9fec3e172a863e7d46cee63de1575"
server_key_version: "encryptionkey@0"
I20260619 01:06:22.777987 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:22.796836 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:22.801904 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:22.802197 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:22.805102 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:22.805233 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:22.805392 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:22.805472 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:06:22 dist-test-slave-5p8w krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:22.831638 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:22.805971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45243 (local address 127.0.97.62:41439)
0619 01:06:22.806146 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:22.806154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:22.807737 (+ 1583us) server_negotiation.cc:407] Connection header received
0619 01:06:22.810705 (+ 2968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:22.810711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:22.810807 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:22.810932 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:22.812726 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.813941 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.816613 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.817272 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.817466 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:22.823432 (+ 5966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:22.823442 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:22.823450 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:22.823506 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:22.827050 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:22.828200 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:22.828210 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:22.828223 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:22.828410 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:22.829195 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:22.829202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:22.829206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:22.829716 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:22.829880 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:22.831041 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0619 01:06:22.831264 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:06:22.833420 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:22.806597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41439 (local address 127.0.97.2:45243)
0619 01:06:22.807520 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:22.807572 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:22.810286 (+ 2714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:22.811196 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:22.811204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:22.811873 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:22.812534 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:22.812563 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.814116 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.814124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:22.816439 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:22.816456 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.817614 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.817626 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:22.817982 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:22.819029 (+ 1047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:22.819071 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:22.823091 (+ 4020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:22.827224 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:22.827234 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:22.827263 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:22.828017 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:22.828597 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:22.828605 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:22.828612 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:22.828979 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:22.829911 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:22.829923 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:22.830508 (+ 585us) client_negotiation.cc:769] Sending connection context
0619 01:06:22.830946 (+ 438us) client_negotiation.cc:240] Negotiation successful
0619 01:06:22.831574 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":402,"threads_started":1}
I20260619 01:06:22.850602 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38405
I20260619 01:06:22.850623 1858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38405 every 8 connection(s)
I20260619 01:06:22.851938 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:06:22.852712 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 1724
I20260619 01:06:22.852932 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:06:22.853646 388 external_mini_cluster.cc:1428] Setting key 9c869a24125980a5416160a2aa649e96
I20260619 01:06:22.854615 1859 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41439
I20260619 01:06:22.854875 1859 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:22.855798 1859 heartbeater.cc:511] Master 127.0.97.62:41439 requested a full tablet report, sending...
I20260619 01:06:22.857309 1513 ts_manager.cc:194] Registered new tserver with Master: 15fa75bc2c5e41ca8c3881ac92ba6352 (127.0.97.2:38405)
I20260619 01:06:22.857600 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:22.841757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41439 (local address 127.0.97.2:52075)
0619 01:06:22.842405 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:22.842426 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:22.842653 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:22.843194 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:22.843201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:22.843903 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:22.844388 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:22.844406 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.845891 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.845899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:22.847732 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:22.847752 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.848891 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.848895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:22.849015 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:22.849898 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:22.849916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:22.850780 (+ 864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:22.854412 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:22.854417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:22.854421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:22.855036 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:22.855517 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:22.855522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:22.855525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:22.855681 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:22.856423 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:22.856428 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:22.856626 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 01:06:22.856748 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 01:06:22.857098 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":463,"threads_started":1}
I20260619 01:06:22.858433 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45243
I20260619 01:06:22.858665 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:22.841594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52075 (local address 127.0.97.62:41439)
0619 01:06:22.842209 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:22.842219 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:22.842518 (+ 299us) server_negotiation.cc:407] Connection header received
0619 01:06:22.842828 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:22.842834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:22.842910 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:22.842968 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:22.844620 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.845659 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.847968 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:22.848572 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:22.848652 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:22.850977 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:22.850982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:22.850985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:22.851019 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:22.854244 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:22.855220 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:22.855224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:22.855228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:22.855382 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:22.855821 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:22.855828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:22.855833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:22.856272 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:22.857963 (+ 1691us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:22.858078 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 01:06:22.858289 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:06:22.897073 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41439
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:23.238796 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:23.239099 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:23.239257 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:23.247184 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:23.247319 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:23.247473 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:06:23.247607 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:06:23.258862 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:23.260349 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:23.262006 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:23.277261 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:23.277340 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:23.277375 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:23.278261 1863 server_base.cc:1034] running on GCE node
I20260619 01:06:23.279088 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:23.280187 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:23.281428 1863 hybrid_clock.cc:648] HybridClock initialized: now 1781831183281342 us; error 67 us; skew 500 ppm
I20260619 01:06:23.281888 1863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:23 dist-test-slave-5p8w krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:23.285667 1863 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:06:23.287343 1863 webserver.cc:460] Webserver started at http://127.0.97.3:35519/ using document root <none> and password file <none>
I20260619 01:06:23.288121 1863 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:23.288261 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:23.288617 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:23.291953 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "bcbaee20fdc943f382a2f3f86fb47c91"
format_stamp: "Formatted at 2026-06-19 01:06:23 on dist-test-slave-5p8w"
server_key: "25eac4815be9d44fc2da23e8ef1a2f23"
server_key_iv: "f12c486c239a67cd93710dcb3983ef76"
server_key_version: "encryptionkey@0"
I20260619 01:06:23.292884 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bcbaee20fdc943f382a2f3f86fb47c91"
format_stamp: "Formatted at 2026-06-19 01:06:23 on dist-test-slave-5p8w"
server_key: "25eac4815be9d44fc2da23e8ef1a2f23"
server_key_iv: "f12c486c239a67cd93710dcb3983ef76"
server_key_version: "encryptionkey@0"
I20260619 01:06:23.298667 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 01:06:23.302816 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:23.304494 1863 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 01:06:23.304705 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "bcbaee20fdc943f382a2f3f86fb47c91"
format_stamp: "Formatted at 2026-06-19 01:06:23 on dist-test-slave-5p8w"
server_key: "25eac4815be9d44fc2da23e8ef1a2f23"
server_key_iv: "f12c486c239a67cd93710dcb3983ef76"
server_key_version: "encryptionkey@0"
I20260619 01:06:23.304971 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:23.325892 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:23.331522 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:23.331758 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:23.334780 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:23.334895 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:23.335040 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:23.335109 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:23.336376 1720 heartbeater.cc:503] Master 127.0.97.62:41439 was elected leader, sending a full tablet report...
Jun 19 01:06:23 dist-test-slave-5p8w krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:23.363632 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.335267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40527 (local address 127.0.97.62:41439)
0619 01:06:23.335528 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.335534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.337171 (+ 1637us) server_negotiation.cc:407] Connection header received
0619 01:06:23.340288 (+ 3117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.340296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.340403 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.340544 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:23.343536 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.344849 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.348475 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.349085 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.349203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.355536 (+ 6333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:23.355542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:23.355545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:23.355585 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:23.359574 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:23.360665 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:23.360673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:23.360681 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:23.360824 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:23.361340 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:23.361347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:23.361351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:23.361788 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:23.361953 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.363019 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.363260 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 01:06:23.365474 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.335984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41439 (local address 127.0.97.3:40527)
0619 01:06:23.336973 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:23.337040 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:23.339889 (+ 2849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:23.340939 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:23.340956 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:23.342026 (+ 1070us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:23.343264 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.343300 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.345084 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.345098 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.348235 (+ 3137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.348263 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.349452 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.349464 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.349898 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.350870 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:23.350918 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:23.355155 (+ 4237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:23.359781 (+ 4626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:23.359793 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:23.359820 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:23.360511 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:23.360950 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:23.360955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:23.360959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:23.361175 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:23.361944 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:23.361953 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:23.362505 (+ 552us) client_negotiation.cc:769] Sending connection context
0619 01:06:23.362870 (+ 365us) client_negotiation.cc:240] Negotiation successful
0619 01:06:23.363525 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":306,"threads_started":1}
I20260619 01:06:23.379798 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45927
I20260619 01:06:23.379889 1997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45927 every 8 connection(s)
I20260619 01:06:23.381242 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:06:23.383641 1998 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41439
I20260619 01:06:23.383889 1998 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:23.384759 1998 heartbeater.cc:511] Master 127.0.97.62:41439 requested a full tablet report, sending...
I20260619 01:06:23.386437 1513 ts_manager.cc:194] Registered new tserver with Master: bcbaee20fdc943f382a2f3f86fb47c91 (127.0.97.3:45927)
I20260619 01:06:23.387205 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 1863
I20260619 01:06:23.387380 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:06:23.387558 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40527
I20260619 01:06:23.387629 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.372658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33823 (local address 127.0.97.62:41439)
0619 01:06:23.372844 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.372849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.373812 (+ 963us) server_negotiation.cc:407] Connection header received
0619 01:06:23.374085 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.374091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.374180 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.374300 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:23.375791 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.376683 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.378372 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.378877 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.379016 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.381196 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:23.381201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:23.381205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:23.381237 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:23.384737 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:23.385515 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:23.385520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:23.385524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:23.385686 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:23.386059 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:23.386064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:23.386066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:23.386387 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:23.386552 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.387004 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.387234 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:06:23.387689 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.372778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41439 (local address 127.0.97.3:33823)
0619 01:06:23.373677 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:23.373706 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:23.373934 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:23.374335 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:23.374342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:23.375048 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:23.375558 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.375577 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.376881 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.376884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.378211 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.378223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.379261 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.379266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.379415 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.380231 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:23.380251 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:23.380983 (+ 732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:23.384892 (+ 3909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:23.384898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:23.384904 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:23.385394 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:23.385800 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:23.385806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:23.385809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:23.385946 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:23.386513 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:23.386519 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:23.386710 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 01:06:23.386866 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 01:06:23.387217 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":369,"threads_started":1}
I20260619 01:06:23.387903 388 external_mini_cluster.cc:1428] Setting key 0fc0eeab71c3fe65e8f009c2c5300509
I20260619 01:06:23.390955 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 01:06:23 dist-test-slave-5p8w krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831183, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 01:06:23 dist-test-slave-5p8w krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831183, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:23.441846 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.427446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.0.97.62:41439)
0619 01:06:23.427688 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.427695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.427920 (+ 225us) server_negotiation.cc:407] Connection header received
0619 01:06:23.428140 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.428145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.428218 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.428340 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:23.429903 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.430638 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.431990 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.432409 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.432553 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.436592 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:23.436597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:23.436600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:23.436639 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:23.439159 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:23.439844 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:23.439848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:23.439850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:23.439968 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:23.440366 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:23.440372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:23.440375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:23.440639 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:23.440933 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.441185 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.441451 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 01:06:23.445878 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:06:23.453016 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:06:23.488574 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.466411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38405 (local address 127.0.0.1:58432)
0619 01:06:23.479806 (+ 13395us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:23.479865 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:23.480062 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:23.480947 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:23.480956 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:23.480995 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:23.481507 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.481518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.483394 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.483401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.486012 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.486023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.487477 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.487483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.487698 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.487770 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 01:06:23.487888 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 01:06:23.488096 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13083,"mutex_wait_us":23,"thread_start_us":14225,"threads_started":1}
I20260619 01:06:23.488801 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.466624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.0.97.2:38405)
0619 01:06:23.480131 (+ 13507us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.480144 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.480187 (+ 43us) server_negotiation.cc:407] Connection header received
0619 01:06:23.480344 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.480357 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.480691 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.480929 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:23.481753 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.483256 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.486190 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.487228 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.487938 (+ 710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.488196 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.488308 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.488494 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13273,"thread_start_us":13648,"threads_started":1}
I20260619 01:06:23.489966 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.467225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.0.97.3:45927)
0619 01:06:23.478960 (+ 11735us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.478970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.481047 (+ 2077us) server_negotiation.cc:407] Connection header received
0619 01:06:23.481407 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.481414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.481694 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.481877 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:23.482969 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.484500 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.487044 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.488607 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.488735 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.488987 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.489494 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.489649 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11508,"thread_start_us":11753,"threads_started":1}
I20260619 01:06:23.489975 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.466702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45927 (local address 127.0.0.1:40908)
0619 01:06:23.480929 (+ 14227us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:23.480946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:23.481168 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:23.482192 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:23.482200 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:23.482240 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:23.482741 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.482756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.484666 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.484671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.486857 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.486871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.489117 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.489121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.489235 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.489258 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 01:06:23.489436 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 01:06:23.489574 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14077,"mutex_wait_us":1,"spinlock_wait_cycles":11008,"thread_start_us":13247,"threads_started":1}
I20260619 01:06:23.491494 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.482021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38513 (local address 127.0.0.1:42832)
0619 01:06:23.482709 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:23.482727 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:23.482859 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:23.483467 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:23.483471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:23.483507 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:23.483959 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.483970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.485855 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.485860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.487718 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.487729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.490667 (+ 2938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.490675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.490864 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.490901 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 01:06:23.490999 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 01:06:23.491143 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":1,"thread_start_us":320,"threads_started":1}
I20260619 01:06:23.491555 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.481388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.0.97.1:38513)
0619 01:06:23.481970 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.481978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.482833 (+ 855us) server_negotiation.cc:407] Connection header received
0619 01:06:23.482996 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.483006 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.483277 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.483474 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:23.484173 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.485705 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.487874 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.489414 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.489533 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.489717 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.491069 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.491259 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":271,"threads_started":1}
I20260619 01:06:23.493234 1789 tablet_service.cc:1467] Processing CreateTablet for tablet cdc41e9dabe642f0a38dc4d97135e8a4 (DEFAULT_TABLE table=test-table [id=d0270d5271e04866aca50891bc7b6b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:23.493634 1928 tablet_service.cc:1467] Processing CreateTablet for tablet cdc41e9dabe642f0a38dc4d97135e8a4 (DEFAULT_TABLE table=test-table [id=d0270d5271e04866aca50891bc7b6b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:23.494850 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc41e9dabe642f0a38dc4d97135e8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:23.494930 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc41e9dabe642f0a38dc4d97135e8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:23.495538 1650 tablet_service.cc:1467] Processing CreateTablet for tablet cdc41e9dabe642f0a38dc4d97135e8a4 (DEFAULT_TABLE table=test-table [id=d0270d5271e04866aca50891bc7b6b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:23.496948 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc41e9dabe642f0a38dc4d97135e8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:23.506537 2017 tablet_bootstrap.cc:492] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: Bootstrap starting.
I20260619 01:06:23.510916 2017 tablet_bootstrap.cc:654] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:23.512135 2017 log.cc:824] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:23.514765 2020 tablet_bootstrap.cc:492] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: Bootstrap starting.
I20260619 01:06:23.513677 2018 tablet_bootstrap.cc:492] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: Bootstrap starting.
I20260619 01:06:23.515581 2017 tablet_bootstrap.cc:492] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: No bootstrap required, opened a new log
I20260619 01:06:23.515847 2017 ts_tablet_manager.cc:1397] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260619 01:06:23.518573 2020 tablet_bootstrap.cc:654] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:23.518573 2018 tablet_bootstrap.cc:654] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:23.519917 2020 log.cc:824] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:23.520550 2018 log.cc:824] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:23.523945 2018 tablet_bootstrap.cc:492] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: No bootstrap required, opened a new log
I20260619 01:06:23.524209 2020 tablet_bootstrap.cc:492] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: No bootstrap required, opened a new log
I20260619 01:06:23.524268 2018 ts_tablet_manager.cc:1397] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20260619 01:06:23.524516 2020 ts_tablet_manager.cc:1397] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 01:06:23.524884 2017 raft_consensus.cc:348] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } }
I20260619 01:06:23.525414 2017 raft_consensus.cc:374] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:23.525521 2017 raft_consensus.cc:729] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbaee20fdc943f382a2f3f86fb47c91, State: Initialized, Role: FOLLOWER
I20260619 01:06:23.526199 2017 consensus_queue.cc:260] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } }
I20260619 01:06:23.527837 1998 heartbeater.cc:503] Master 127.0.97.62:41439 was elected leader, sending a full tablet report...
I20260619 01:06:23.528642 2017 ts_tablet_manager.cc:1428] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260619 01:06:23.533985 2020 raft_consensus.cc:348] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } }
I20260619 01:06:23.534407 2020 raft_consensus.cc:374] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:23.534987 2020 raft_consensus.cc:729] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7dd1c7602749e3ad4c612bda902028, State: Initialized, Role: FOLLOWER
I20260619 01:06:23.535714 2020 consensus_queue.cc:260] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } }
I20260619 01:06:23.536230 2018 raft_consensus.cc:348] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } }
I20260619 01:06:23.536640 2018 raft_consensus.cc:374] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:23.536753 2018 raft_consensus.cc:729] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15fa75bc2c5e41ca8c3881ac92ba6352, State: Initialized, Role: FOLLOWER
I20260619 01:06:23.537569 2020 ts_tablet_manager.cc:1428] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 01:06:23.537395 2018 consensus_queue.cc:260] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } }
I20260619 01:06:23.538421 1859 heartbeater.cc:503] Master 127.0.97.62:41439 was elected leader, sending a full tablet report...
I20260619 01:06:23.539007 2018 ts_tablet_manager.cc:1428] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
W20260619 01:06:23.578172 1721 tablet.cc:2369] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:06:23.610482 1860 tablet.cc:2369] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:06:23.633322 1999 tablet.cc:2369] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:06:23.717024 2023 raft_consensus.cc:482] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:23.717427 2023 raft_consensus.cc:504] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } }
I20260619 01:06:23.719208 2023 leader_election.cc:288] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15fa75bc2c5e41ca8c3881ac92ba6352 (127.0.97.2:38405), fc7dd1c7602749e3ad4c612bda902028 (127.0.97.1:38513)
I20260619 01:06:23.727262 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.720245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37737 (local address 127.0.97.1:38513)
0619 01:06:23.720508 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.720516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.720894 (+ 378us) server_negotiation.cc:407] Connection header received
0619 01:06:23.721052 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.721055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.721119 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.721227 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:23.721764 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.722662 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.724987 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.726286 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.726405 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.726502 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.726860 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.726994 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 01:06:23.727275 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.720259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46145 (local address 127.0.97.2:38405)
0619 01:06:23.720523 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.720529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.720550 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:06:23.720624 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.720630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.720712 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.720801 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:23.721251 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.722309 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.724896 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.726173 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.726347 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.726494 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.726860 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.726994 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:06:23.727412 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.719716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38405 (local address 127.0.97.3:46145)
0619 01:06:23.719931 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:23.719950 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:23.720143 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:23.720853 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:23.720857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:23.720888 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:23.721105 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.721113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.722462 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.722467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.724712 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.724729 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.726343 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.726370 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.726636 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.726713 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 01:06:23.726816 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 01:06:23.727033 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":135}
I20260619 01:06:23.727487 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.720097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38513 (local address 127.0.97.3:37737)
0619 01:06:23.720746 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:23.720763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:23.720912 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:23.721260 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:23.721262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:23.721280 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:23.721602 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.721610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.722801 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.722806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.724821 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.724835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.726497 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.726503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.726696 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.726721 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 01:06:23.726816 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 01:06:23.727033 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":46,"spinlock_wait_cycles":14592,"thread_start_us":386,"threads_started":1}
I20260619 01:06:23.728515 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc41e9dabe642f0a38dc4d97135e8a4" candidate_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7dd1c7602749e3ad4c612bda902028" is_pre_election: true
I20260619 01:06:23.728637 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc41e9dabe642f0a38dc4d97135e8a4" candidate_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" is_pre_election: true
I20260619 01:06:23.729075 1670 raft_consensus.cc:2454] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcbaee20fdc943f382a2f3f86fb47c91 in term 0.
I20260619 01:06:23.729148 1809 raft_consensus.cc:2454] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcbaee20fdc943f382a2f3f86fb47c91 in term 0.
I20260619 01:06:23.729919 1882 leader_election.cc:302] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcbaee20fdc943f382a2f3f86fb47c91, fc7dd1c7602749e3ad4c612bda902028; no voters:
I20260619 01:06:23.730360 2023 raft_consensus.cc:2781] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:06:23.730526 2023 raft_consensus.cc:482] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:06:23.730635 2023 raft_consensus.cc:3037] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:23.732897 2023 raft_consensus.cc:504] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } }
I20260619 01:06:23.733628 2023 leader_election.cc:288] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [CANDIDATE]: Term 1 election: Requested vote from peers 15fa75bc2c5e41ca8c3881ac92ba6352 (127.0.97.2:38405), fc7dd1c7602749e3ad4c612bda902028 (127.0.97.1:38513)
I20260619 01:06:23.734222 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc41e9dabe642f0a38dc4d97135e8a4" candidate_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352"
I20260619 01:06:23.734302 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc41e9dabe642f0a38dc4d97135e8a4" candidate_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7dd1c7602749e3ad4c612bda902028"
I20260619 01:06:23.734486 1809 raft_consensus.cc:3037] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:23.734543 1670 raft_consensus.cc:3037] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:23.737161 1809 raft_consensus.cc:2454] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcbaee20fdc943f382a2f3f86fb47c91 in term 1.
I20260619 01:06:23.737361 1670 raft_consensus.cc:2454] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcbaee20fdc943f382a2f3f86fb47c91 in term 1.
I20260619 01:06:23.737753 1882 leader_election.cc:302] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15fa75bc2c5e41ca8c3881ac92ba6352, bcbaee20fdc943f382a2f3f86fb47c91; no voters:
I20260619 01:06:23.738050 2023 raft_consensus.cc:2781] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:23.738559 2023 raft_consensus.cc:686] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 LEADER]: Becoming Leader. State: Replica: bcbaee20fdc943f382a2f3f86fb47c91, State: Running, Role: LEADER
I20260619 01:06:23.739049 2023 consensus_queue.cc:237] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } }
I20260619 01:06:23.744345 1512 catalog_manager.cc:5495] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcbaee20fdc943f382a2f3f86fb47c91 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } health_report { overall_health: UNKNOWN } } }
I20260619 01:06:23.784580 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.778736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.0.97.3:45927)
0619 01:06:23.778973 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.778977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.778996 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:06:23.779264 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.779267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.779355 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.779478 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:23.779895 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.780607 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.782265 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.782753 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.782911 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.783387 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:23.783958 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:23.784070 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.784199 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.784269 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:06:23.793668 1809 raft_consensus.cc:1261] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 1 FOLLOWER]: Refusing update from remote peer bcbaee20fdc943f382a2f3f86fb47c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:23.793691 1670 raft_consensus.cc:1261] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028 [term 1 FOLLOWER]: Refusing update from remote peer bcbaee20fdc943f382a2f3f86fb47c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:23.794711 2027 consensus_queue.cc:1035] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:23.795212 2023 consensus_queue.cc:1035] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:23.809461 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298380528808058880 to 7298380528599158784. Current Snapshot: MvccSnapshot[applied={T|T < 7298380528808058880}]
I20260619 01:06:23.836005 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:23.836310 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:23.840157 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b: Sending DeleteTablet for 3 replicas of tablet cdc41e9dabe642f0a38dc4d97135e8a4
I20260619 01:06:23.841604 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc41e9dabe642f0a38dc4d97135e8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40908
I20260619 01:06:23.841908 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc41e9dabe642f0a38dc4d97135e8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42832
I20260619 01:06:23.842327 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc41e9dabe642f0a38dc4d97135e8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58432
I20260619 01:06:23.843086 2040 tablet_replica.cc:330] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: stopping tablet replica
I20260619 01:06:23.843225 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 01:06:23.843561 2040 raft_consensus.cc:2229] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 01:06:23.843900 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:06:23.844072 2040 raft_consensus.cc:2258] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:23.846698 2040 ts_tablet_manager.cc:1905] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:23.850562 2042 tablet_replica.cc:330] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: stopping tablet replica
I20260619 01:06:23.851135 2042 raft_consensus.cc:2229] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:06:23.852017 2042 raft_consensus.cc:2258] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:23.854127 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed7292af36d4751bf5f0830ed8b2530 (DEFAULT_TABLE table=test-table [id=625564d90a4e4f25bd02453e3f0e7d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:23.854378 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed7292af36d4751bf5f0830ed8b2530 (DEFAULT_TABLE table=test-table [id=625564d90a4e4f25bd02453e3f0e7d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:23.854624 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed7292af36d4751bf5f0830ed8b2530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:23.854782 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed7292af36d4751bf5f0830ed8b2530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:23.857926 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed7292af36d4751bf5f0830ed8b2530 (DEFAULT_TABLE table=test-table [id=625564d90a4e4f25bd02453e3f0e7d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:23.858562 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed7292af36d4751bf5f0830ed8b2530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:23.859432 2042 ts_tablet_manager.cc:1905] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:23.863734 2017 tablet_bootstrap.cc:492] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91: Bootstrap starting.
I20260619 01:06:23.865628 2017 tablet_bootstrap.cc:654] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:23.867151 2018 tablet_bootstrap.cc:492] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352: Bootstrap starting.
I20260619 01:06:23.867432 2042 ts_tablet_manager.cc:1918] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:06:23.868326 2040 ts_tablet_manager.cc:1918] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:06:23.869421 2018 tablet_bootstrap.cc:654] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:23.869652 2040 log.cc:1196] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/cdc41e9dabe642f0a38dc4d97135e8a4
I20260619 01:06:23.870306 2040 ts_tablet_manager.cc:1939] T cdc41e9dabe642f0a38dc4d97135e8a4 P fc7dd1c7602749e3ad4c612bda902028: Deleting consensus metadata
I20260619 01:06:23.873661 2020 tablet_bootstrap.cc:492] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028: Bootstrap starting.
I20260619 01:06:23.875345 2020 tablet_bootstrap.cc:654] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:23.877404 2042 log.cc:1196] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/cdc41e9dabe642f0a38dc4d97135e8a4
I20260619 01:06:23.877367 1500 catalog_manager.cc:4841] TS fc7dd1c7602749e3ad4c612bda902028 (127.0.97.1:38513): tablet cdc41e9dabe642f0a38dc4d97135e8a4 (table test-table [id=d0270d5271e04866aca50891bc7b6b89]) successfully deleted
I20260619 01:06:23.878082 2042 ts_tablet_manager.cc:1939] T cdc41e9dabe642f0a38dc4d97135e8a4 P bcbaee20fdc943f382a2f3f86fb47c91: Deleting consensus metadata
I20260619 01:06:23.880650 1498 catalog_manager.cc:4841] TS bcbaee20fdc943f382a2f3f86fb47c91 (127.0.97.3:45927): tablet cdc41e9dabe642f0a38dc4d97135e8a4 (table test-table [id=d0270d5271e04866aca50891bc7b6b89]) successfully deleted
I20260619 01:06:23.889814 2020 tablet_bootstrap.cc:492] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028: No bootstrap required, opened a new log
I20260619 01:06:23.890027 2020 ts_tablet_manager.cc:1397] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260619 01:06:23.890857 2020 raft_consensus.cc:348] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } }
I20260619 01:06:23.891125 2020 raft_consensus.cc:374] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:23.891232 2020 raft_consensus.cc:729] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7dd1c7602749e3ad4c612bda902028, State: Initialized, Role: FOLLOWER
I20260619 01:06:23.891497 2020 consensus_queue.cc:260] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } }
I20260619 01:06:23.892132 2020 ts_tablet_manager.cc:1428] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:06:23.892163 2018 tablet_bootstrap.cc:492] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352: No bootstrap required, opened a new log
I20260619 01:06:23.892329 2018 ts_tablet_manager.cc:1397] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.005s
I20260619 01:06:23.893163 2018 raft_consensus.cc:348] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } }
I20260619 01:06:23.893410 2018 raft_consensus.cc:374] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:23.893537 2018 raft_consensus.cc:729] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15fa75bc2c5e41ca8c3881ac92ba6352, State: Initialized, Role: FOLLOWER
I20260619 01:06:23.893797 2018 consensus_queue.cc:260] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } }
I20260619 01:06:23.894399 2018 ts_tablet_manager.cc:1428] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:06:23.896474 2017 tablet_bootstrap.cc:492] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91: No bootstrap required, opened a new log
I20260619 01:06:23.896656 2017 ts_tablet_manager.cc:1397] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.006s
I20260619 01:06:23.897509 2017 raft_consensus.cc:348] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } }
I20260619 01:06:23.897773 2017 raft_consensus.cc:374] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:23.897887 2017 raft_consensus.cc:729] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbaee20fdc943f382a2f3f86fb47c91, State: Initialized, Role: FOLLOWER
I20260619 01:06:23.898169 2017 consensus_queue.cc:260] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } }
I20260619 01:06:23.899088 2017 ts_tablet_manager.cc:1428] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:06:23.899758 2041 tablet_replica.cc:330] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: stopping tablet replica
I20260619 01:06:23.900282 2041 raft_consensus.cc:2229] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:23.900785 2041 raft_consensus.cc:2258] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:23.903052 2041 ts_tablet_manager.cc:1905] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:23.906834 2024 raft_consensus.cc:482] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:23.907080 2024 raft_consensus.cc:504] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } }
I20260619 01:06:23.908440 2041 ts_tablet_manager.cc:1918] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:06:23.908628 2041 log.cc:1196] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/cdc41e9dabe642f0a38dc4d97135e8a4
I20260619 01:06:23.909216 2041 ts_tablet_manager.cc:1939] T cdc41e9dabe642f0a38dc4d97135e8a4 P 15fa75bc2c5e41ca8c3881ac92ba6352: Deleting consensus metadata
I20260619 01:06:23.910861 2024 leader_election.cc:288] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15fa75bc2c5e41ca8c3881ac92ba6352 (127.0.97.2:38405), bcbaee20fdc943f382a2f3f86fb47c91 (127.0.97.3:45927)
I20260619 01:06:23.911355 1500 catalog_manager.cc:4841] TS 15fa75bc2c5e41ca8c3881ac92ba6352 (127.0.97.2:38405): tablet cdc41e9dabe642f0a38dc4d97135e8a4 (table test-table [id=d0270d5271e04866aca50891bc7b6b89]) successfully deleted
I20260619 01:06:23.921239 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.910573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59743 (local address 127.0.97.3:45927)
0619 01:06:23.910760 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.910766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.914283 (+ 3517us) server_negotiation.cc:407] Connection header received
0619 01:06:23.914485 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.914490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.914564 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.914668 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:23.915552 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.916650 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.918584 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.920101 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.920233 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.920343 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.920733 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.920903 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:06:23.921267 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.910176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45927 (local address 127.0.97.1:59743)
0619 01:06:23.914191 (+ 4015us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:23.914208 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:23.914319 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:23.914711 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:23.914716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:23.914744 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:23.915378 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.915390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.916784 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.916789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.918400 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.918415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.920342 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.920350 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.920542 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.920579 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 01:06:23.920688 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 01:06:23.920843 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3886,"thread_start_us":3713,"threads_started":1}
I20260619 01:06:23.922333 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.910446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33653 (local address 127.0.97.2:38405)
0619 01:06:23.910623 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.910630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.913954 (+ 3324us) server_negotiation.cc:407] Connection header received
0619 01:06:23.914269 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.914276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.914369 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.914480 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:23.915255 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.916400 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.919603 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.920718 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.920847 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.920936 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.921899 (+ 963us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.922036 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:06:23.922209 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed7292af36d4751bf5f0830ed8b2530" candidate_uuid: "fc7dd1c7602749e3ad4c612bda902028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" is_pre_election: true
I20260619 01:06:23.922678 1948 raft_consensus.cc:2454] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc7dd1c7602749e3ad4c612bda902028 in term 0.
I20260619 01:06:23.922938 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.910020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38405 (local address 127.0.97.1:33653)
0619 01:06:23.913828 (+ 3808us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:23.913855 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:23.914092 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:23.914516 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:23.914520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:23.914551 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:23.915064 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.915080 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.916556 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.916563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.918955 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.918971 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.921098 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.921104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.921345 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.921738 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 01:06:23.921842 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 01:06:23.921990 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3647,"mutex_wait_us":2,"thread_start_us":3482,"threads_started":1}
I20260619 01:06:23.923457 1601 leader_election.cc:302] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcbaee20fdc943f382a2f3f86fb47c91, fc7dd1c7602749e3ad4c612bda902028; no voters:
I20260619 01:06:23.923636 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed7292af36d4751bf5f0830ed8b2530" candidate_uuid: "fc7dd1c7602749e3ad4c612bda902028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" is_pre_election: true
I20260619 01:06:23.923918 2024 raft_consensus.cc:2781] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:06:23.923905 1809 raft_consensus.cc:2454] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc7dd1c7602749e3ad4c612bda902028 in term 0.
I20260619 01:06:23.924048 2024 raft_consensus.cc:482] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:06:23.924137 2024 raft_consensus.cc:3037] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:23.927438 2024 raft_consensus.cc:504] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } }
I20260619 01:06:23.928241 2024 leader_election.cc:288] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [CANDIDATE]: Term 1 election: Requested vote from peers 15fa75bc2c5e41ca8c3881ac92ba6352 (127.0.97.2:38405), bcbaee20fdc943f382a2f3f86fb47c91 (127.0.97.3:45927)
I20260619 01:06:23.928836 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed7292af36d4751bf5f0830ed8b2530" candidate_uuid: "fc7dd1c7602749e3ad4c612bda902028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352"
I20260619 01:06:23.928864 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed7292af36d4751bf5f0830ed8b2530" candidate_uuid: "fc7dd1c7602749e3ad4c612bda902028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbaee20fdc943f382a2f3f86fb47c91"
I20260619 01:06:23.929085 1809 raft_consensus.cc:3037] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:23.929091 1948 raft_consensus.cc:3037] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:23.932210 1809 raft_consensus.cc:2454] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc7dd1c7602749e3ad4c612bda902028 in term 1.
I20260619 01:06:23.932786 1948 raft_consensus.cc:2454] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc7dd1c7602749e3ad4c612bda902028 in term 1.
I20260619 01:06:23.932837 1603 leader_election.cc:302] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15fa75bc2c5e41ca8c3881ac92ba6352, fc7dd1c7602749e3ad4c612bda902028; no voters:
I20260619 01:06:23.933228 2024 raft_consensus.cc:2781] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:23.933831 2024 raft_consensus.cc:686] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 1 LEADER]: Becoming Leader. State: Replica: fc7dd1c7602749e3ad4c612bda902028, State: Running, Role: LEADER
I20260619 01:06:23.934269 2024 consensus_queue.cc:237] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } }
I20260619 01:06:23.938735 1511 catalog_manager.cc:5495] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc7dd1c7602749e3ad4c612bda902028 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fc7dd1c7602749e3ad4c612bda902028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } health_report { overall_health: UNKNOWN } } }
Jun 19 01:06:23 dist-test-slave-5p8w krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:23.994004 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.975939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41439 (local address 127.0.0.1:47408)
0619 01:06:23.976918 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:23.976941 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:23.977142 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:23.977555 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:23.977558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:23.978734 (+ 1176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:23.979205 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.979220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.980811 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.980815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.982509 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:23.982521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.983537 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.983541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:23.983693 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.984459 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:23.984477 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:23.987129 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:23.991236 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:23.991261 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:23.991267 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:23.991696 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:23.992181 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:23.992186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:23.992190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:23.992404 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:23.993059 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:23.993070 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:23.993258 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 01:06:23.993454 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 01:06:23.993720 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":622,"threads_started":1}
I20260619 01:06:23.994501 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:23.975939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.0.97.62:41439)
0619 01:06:23.976661 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:23.976668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:23.977062 (+ 394us) server_negotiation.cc:407] Connection header received
0619 01:06:23.977282 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:23.977305 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:23.977392 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:23.977523 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:23.979415 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.980645 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.982659 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:23.983352 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:23.983476 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:23.987379 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:23.987387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:23.987393 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:23.987432 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:23.991069 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:23.991817 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:23.991823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:23.991827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:23.992002 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:23.992519 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:23.992525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:23.992528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:23.992903 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:23.993071 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:23.993863 (+ 792us) server_negotiation.cc:299] Negotiation successful
0619 01:06:23.994111 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":346,"threads_started":1}
I20260619 01:06:23.999938 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47408:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 01:06:24.001019 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:06:24.008054 1650 tablet_service.cc:1467] Processing CreateTablet for tablet deb6e18166984f2c881d756c326149b6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=653c11fa6c604bf1ac02e2e93a64b580]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:06:24.008208 1928 tablet_service.cc:1467] Processing CreateTablet for tablet deb6e18166984f2c881d756c326149b6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=653c11fa6c604bf1ac02e2e93a64b580]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:06:24.008368 1789 tablet_service.cc:1467] Processing CreateTablet for tablet deb6e18166984f2c881d756c326149b6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=653c11fa6c604bf1ac02e2e93a64b580]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:06:24.008698 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb6e18166984f2c881d756c326149b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:24.008836 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb6e18166984f2c881d756c326149b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:24.008924 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb6e18166984f2c881d756c326149b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:24.017131 2020 tablet_bootstrap.cc:492] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028: Bootstrap starting.
I20260619 01:06:24.017462 2018 tablet_bootstrap.cc:492] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352: Bootstrap starting.
I20260619 01:06:24.018931 2020 tablet_bootstrap.cc:654] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:24.019110 2018 tablet_bootstrap.cc:654] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:24.020876 2017 tablet_bootstrap.cc:492] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: Bootstrap starting.
I20260619 01:06:24.023190 2017 tablet_bootstrap.cc:654] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:24.029397 2020 tablet_bootstrap.cc:492] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028: No bootstrap required, opened a new log
I20260619 01:06:24.029587 2020 ts_tablet_manager.cc:1397] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260619 01:06:24.030397 2020 raft_consensus.cc:348] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } }
I20260619 01:06:24.030661 2020 raft_consensus.cc:374] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:24.030750 2020 raft_consensus.cc:729] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7dd1c7602749e3ad4c612bda902028, State: Initialized, Role: FOLLOWER
I20260619 01:06:24.031033 2020 consensus_queue.cc:260] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } }
I20260619 01:06:24.031487 2024 tablet_replica.cc:439] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } }
I20260619 01:06:24.031653 2018 tablet_bootstrap.cc:492] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352: No bootstrap required, opened a new log
I20260619 01:06:24.031773 2024 tablet_replica.cc:442] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:24.031847 2018 ts_tablet_manager.cc:1397] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20260619 01:06:24.032889 2018 raft_consensus.cc:348] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } }
I20260619 01:06:24.033206 2018 raft_consensus.cc:374] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:24.033308 2018 raft_consensus.cc:729] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15fa75bc2c5e41ca8c3881ac92ba6352, State: Initialized, Role: FOLLOWER
I20260619 01:06:24.033708 2018 consensus_queue.cc:260] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } }
I20260619 01:06:24.034855 2020 ts_tablet_manager.cc:1428] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 01:06:24.034291 2025 tablet_replica.cc:439] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } }
I20260619 01:06:24.035101 2025 tablet_replica.cc:442] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:24.037955 2018 ts_tablet_manager.cc:1428] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260619 01:06:24.038578 2017 tablet_bootstrap.cc:492] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: No bootstrap required, opened a new log
I20260619 01:06:24.038764 2017 ts_tablet_manager.cc:1397] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.001s
I20260619 01:06:24.039554 2017 raft_consensus.cc:348] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } }
I20260619 01:06:24.039851 2017 raft_consensus.cc:374] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:24.039947 2017 raft_consensus.cc:729] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbaee20fdc943f382a2f3f86fb47c91, State: Initialized, Role: FOLLOWER
I20260619 01:06:24.040210 2017 consensus_queue.cc:260] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } }
I20260619 01:06:24.040823 2017 ts_tablet_manager.cc:1428] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 01:06:24.040755 2023 tablet_replica.cc:439] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } }
I20260619 01:06:24.041039 2023 tablet_replica.cc:442] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:24.058624 2023 raft_consensus.cc:482] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:24.058837 2023 raft_consensus.cc:504] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } }
I20260619 01:06:24.059646 2023 leader_election.cc:288] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc7dd1c7602749e3ad4c612bda902028 (127.0.97.1:38513), 15fa75bc2c5e41ca8c3881ac92ba6352 (127.0.97.2:38405)
I20260619 01:06:24.060374 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb6e18166984f2c881d756c326149b6" candidate_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" is_pre_election: true
I20260619 01:06:24.060364 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb6e18166984f2c881d756c326149b6" candidate_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7dd1c7602749e3ad4c612bda902028" is_pre_election: true
I20260619 01:06:24.060619 1670 raft_consensus.cc:2454] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcbaee20fdc943f382a2f3f86fb47c91 in term 0.
I20260619 01:06:24.060642 1809 raft_consensus.cc:2454] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcbaee20fdc943f382a2f3f86fb47c91 in term 0.
I20260619 01:06:24.061084 1882 leader_election.cc:302] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15fa75bc2c5e41ca8c3881ac92ba6352, bcbaee20fdc943f382a2f3f86fb47c91; no voters:
I20260619 01:06:24.061381 2023 raft_consensus.cc:2781] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:06:24.061512 2023 raft_consensus.cc:482] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:06:24.061616 2023 raft_consensus.cc:3037] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:24.064836 2023 raft_consensus.cc:504] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } }
I20260619 01:06:24.065570 2023 leader_election.cc:288] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [CANDIDATE]: Term 1 election: Requested vote from peers fc7dd1c7602749e3ad4c612bda902028 (127.0.97.1:38513), 15fa75bc2c5e41ca8c3881ac92ba6352 (127.0.97.2:38405)
I20260619 01:06:24.066196 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb6e18166984f2c881d756c326149b6" candidate_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc7dd1c7602749e3ad4c612bda902028"
I20260619 01:06:24.066275 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb6e18166984f2c881d756c326149b6" candidate_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352"
I20260619 01:06:24.066454 1670 raft_consensus.cc:3037] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:24.066491 1809 raft_consensus.cc:3037] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:24.069545 1809 raft_consensus.cc:2454] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcbaee20fdc943f382a2f3f86fb47c91 in term 1.
I20260619 01:06:24.069561 1670 raft_consensus.cc:2454] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcbaee20fdc943f382a2f3f86fb47c91 in term 1.
I20260619 01:06:24.070472 1882 leader_election.cc:302] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15fa75bc2c5e41ca8c3881ac92ba6352, bcbaee20fdc943f382a2f3f86fb47c91; no voters:
I20260619 01:06:24.071072 2023 raft_consensus.cc:2781] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:24.071247 2023 raft_consensus.cc:686] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 LEADER]: Becoming Leader. State: Replica: bcbaee20fdc943f382a2f3f86fb47c91, State: Running, Role: LEADER
I20260619 01:06:24.071549 2023 consensus_queue.cc:237] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } }
I20260619 01:06:24.073544 2035 tablet_replica.cc:439] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: TxnStatusTablet state changed. Reason: New leader bcbaee20fdc943f382a2f3f86fb47c91. Latest consensus state: current_term: 1 leader_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } }
I20260619 01:06:24.073925 2035 tablet_replica.cc:442] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:06:24.074646 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 01:06:24.074837 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 01:06:24.075294 1511 catalog_manager.cc:5495] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcbaee20fdc943f382a2f3f86fb47c91 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } health_report { overall_health: UNKNOWN } } }
I20260619 01:06:24.099081 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.093209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.0.97.3:45927)
0619 01:06:24.093425 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:24.093432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:24.093540 (+ 108us) server_negotiation.cc:407] Connection header received
0619 01:06:24.093847 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:24.093851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:24.093954 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:24.094041 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:24.094468 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.095239 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.097100 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.097692 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.097795 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.098381 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:24.098495 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:24.098579 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:24.098774 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 01:06:24.098842 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:06:24.099223 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.093126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45927 (local address 127.0.0.1:40920)
0619 01:06:24.093416 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:24.093437 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:24.093618 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:24.094095 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:24.094101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:24.094114 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:06:24.094322 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:24.094333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.095377 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.095382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:24.096936 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:24.096952 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.097866 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.097873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:24.098064 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.098144 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:06:24.098610 (+ 466us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:06:24.098634 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 01:06:24.098725 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 01:06:24.098892 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 01:06:24.103181 1670 raft_consensus.cc:1261] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028 [term 1 FOLLOWER]: Refusing update from remote peer bcbaee20fdc943f382a2f3f86fb47c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:24.103253 1809 raft_consensus.cc:1261] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 1 FOLLOWER]: Refusing update from remote peer bcbaee20fdc943f382a2f3f86fb47c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:24.103840 2035 consensus_queue.cc:1035] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:24.104183 2023 consensus_queue.cc:1035] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:24.107432 2024 tablet_replica.cc:439] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028: TxnStatusTablet state changed. Reason: New leader bcbaee20fdc943f382a2f3f86fb47c91. Latest consensus state: current_term: 1 leader_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } }
I20260619 01:06:24.107748 2024 tablet_replica.cc:442] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:24.108135 2025 tablet_replica.cc:439] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352: TxnStatusTablet state changed. Reason: New leader bcbaee20fdc943f382a2f3f86fb47c91. Latest consensus state: current_term: 1 leader_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } }
I20260619 01:06:24.108440 2025 tablet_replica.cc:442] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:24.111541 2035 tablet_replica.cc:439] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } }
I20260619 01:06:24.111826 2035 tablet_replica.cc:442] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:06:24.112943 2027 tablet_replica.cc:439] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } }
I20260619 01:06:24.113698 2027 tablet_replica.cc:442] T deb6e18166984f2c881d756c326149b6 P bcbaee20fdc943f382a2f3f86fb47c91: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:06:24.115056 2032 mvcc.cc:205] Tried to move back new op lower bound from 7298380530080202752 to 7298380529960443904. Current Snapshot: MvccSnapshot[applied={T|T < 7298380530080202752 or (T in {7298380530080202752})}]
I20260619 01:06:24.115477 2024 tablet_replica.cc:439] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } }
I20260619 01:06:24.115769 2024 tablet_replica.cc:442] T deb6e18166984f2c881d756c326149b6 P fc7dd1c7602749e3ad4c612bda902028: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:24.119580 2025 tablet_replica.cc:439] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc7dd1c7602749e3ad4c612bda902028" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38513 } } peers { permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 } } peers { permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 } } }
I20260619 01:06:24.119829 2025 tablet_replica.cc:442] T deb6e18166984f2c881d756c326149b6 P 15fa75bc2c5e41ca8c3881ac92ba6352: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:24.128417 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.121835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.0.97.1:38513)
0619 01:06:24.122037 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:24.122043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:24.122370 (+ 327us) server_negotiation.cc:407] Connection header received
0619 01:06:24.122565 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:24.122570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:24.122664 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:24.122833 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:24.123137 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.124283 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.125787 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.126388 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.126521 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.126978 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:24.127724 (+ 746us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:24.127925 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:24.128010 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 01:06:24.128101 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:06:24.148346 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.141708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45927 (local address 127.0.97.1:48355)
0619 01:06:24.142448 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:24.142466 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:24.142659 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:24.143062 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:24.143065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:24.143074 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:06:24.143322 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:24.143329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.144816 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.144820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:24.146026 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:24.146039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.147119 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.147125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:24.147267 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.147332 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:06:24.147847 (+ 515us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:06:24.147857 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 01:06:24.147929 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 01:06:24.148046 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":393,"threads_started":1}
I20260619 01:06:24.148404 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.141712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48355 (local address 127.0.97.3:45927)
0619 01:06:24.141951 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:24.141958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:24.142552 (+ 594us) server_negotiation.cc:407] Connection header received
0619 01:06:24.142843 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:24.142849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:24.142929 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:24.143037 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:24.143478 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.144679 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.146172 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.146896 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.147046 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.147550 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:24.147727 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:24.147820 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:24.147975 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 01:06:24.148065 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:06:24.159765 1948 raft_consensus.cc:1261] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 FOLLOWER]: Refusing update from remote peer fc7dd1c7602749e3ad4c612bda902028: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:24.160593 1809 raft_consensus.cc:1261] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 1 FOLLOWER]: Refusing update from remote peer fc7dd1c7602749e3ad4c612bda902028: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:24.160807 2048 consensus_queue.cc:1035] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcbaee20fdc943f382a2f3f86fb47c91" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:24.161281 2024 consensus_queue.cc:1035] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15fa75bc2c5e41ca8c3881ac92ba6352" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:24.202600 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.195077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43811 (local address 127.0.97.1:38513)
0619 01:06:24.195265 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:24.195273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:24.195864 (+ 591us) server_negotiation.cc:407] Connection header received
0619 01:06:24.196046 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:24.196051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:24.196153 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:24.196262 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:24.196736 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.197820 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.200131 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.200867 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.201012 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.201753 (+ 741us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:24.201907 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:24.202006 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:24.202176 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 01:06:24.202279 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 01:06:24.203052 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.195082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38513 (local address 127.0.97.3:43811)
0619 01:06:24.195770 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:24.195790 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:24.195908 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:24.196278 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:24.196281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:24.196290 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:06:24.196570 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:24.196577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.197951 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.197957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:24.199956 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:24.199972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.201185 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.201200 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:24.201450 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.201537 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:06:24.202023 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:06:24.202043 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 01:06:24.202146 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 01:06:24.202306 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":382,"threads_started":1}
I20260619 01:06:24.252727 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:24.253007 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:24.256832 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc83b74adcbe4bc4b7b5edde369e170b: Sending DeleteTablet for 3 replicas of tablet 0ed7292af36d4751bf5f0830ed8b2530
I20260619 01:06:24.257925 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed7292af36d4751bf5f0830ed8b2530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58432
I20260619 01:06:24.258136 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed7292af36d4751bf5f0830ed8b2530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40908
I20260619 01:06:24.258294 2041 tablet_replica.cc:330] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352: stopping tablet replica
I20260619 01:06:24.258435 2042 tablet_replica.cc:330] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91: stopping tablet replica
I20260619 01:06:24.258498 2041 raft_consensus.cc:2229] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:24.258687 2041 raft_consensus.cc:2258] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:24.258672 2042 raft_consensus.cc:2229] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:24.258946 2042 raft_consensus.cc:2258] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:24.260602 2041 ts_tablet_manager.cc:1905] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:24.266304 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed7292af36d4751bf5f0830ed8b2530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42832
I20260619 01:06:24.266456 2042 ts_tablet_manager.cc:1905] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:24.269268 2040 tablet_replica.cc:330] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028: stopping tablet replica
I20260619 01:06:24.269562 2040 raft_consensus.cc:2229] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:06:24.270144 2040 raft_consensus.cc:2258] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:24.271097 2041 ts_tablet_manager.cc:1918] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:06:24.271281 2041 log.cc:1196] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/0ed7292af36d4751bf5f0830ed8b2530
I20260619 01:06:24.271411 2040 ts_tablet_manager.cc:1905] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:24.271726 2041 ts_tablet_manager.cc:1939] T 0ed7292af36d4751bf5f0830ed8b2530 P 15fa75bc2c5e41ca8c3881ac92ba6352: Deleting consensus metadata
I20260619 01:06:24.272863 1500 catalog_manager.cc:4841] TS 15fa75bc2c5e41ca8c3881ac92ba6352 (127.0.97.2:38405): tablet 0ed7292af36d4751bf5f0830ed8b2530 (table test-table [id=625564d90a4e4f25bd02453e3f0e7d1d]) successfully deleted
I20260619 01:06:24.274466 2042 ts_tablet_manager.cc:1918] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:06:24.274619 2042 log.cc:1196] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/0ed7292af36d4751bf5f0830ed8b2530
I20260619 01:06:24.275038 2042 ts_tablet_manager.cc:1939] T 0ed7292af36d4751bf5f0830ed8b2530 P bcbaee20fdc943f382a2f3f86fb47c91: Deleting consensus metadata
I20260619 01:06:24.276198 1498 catalog_manager.cc:4841] TS bcbaee20fdc943f382a2f3f86fb47c91 (127.0.97.3:45927): tablet 0ed7292af36d4751bf5f0830ed8b2530 (table test-table [id=625564d90a4e4f25bd02453e3f0e7d1d]) successfully deleted
I20260619 01:06:24.278311 2040 ts_tablet_manager.cc:1918] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:06:24.278498 2040 log.cc:1196] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/0ed7292af36d4751bf5f0830ed8b2530
I20260619 01:06:24.278970 2040 ts_tablet_manager.cc:1939] T 0ed7292af36d4751bf5f0830ed8b2530 P fc7dd1c7602749e3ad4c612bda902028: Deleting consensus metadata
Jun 19 01:06:24 dist-test-slave-5p8w krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831183, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:06:24.280225 1500 catalog_manager.cc:4841] TS fc7dd1c7602749e3ad4c612bda902028 (127.0.97.1:38513): tablet 0ed7292af36d4751bf5f0830ed8b2530 (table test-table [id=625564d90a4e4f25bd02453e3f0e7d1d]) successfully deleted
I20260619 01:06:24.291186 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.265508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.0.97.1:38513)
0619 01:06:24.265712 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:24.265718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:24.265739 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:06:24.265800 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:24.265805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:24.265888 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:24.266007 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:24.269211 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.270511 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.275146 (+ 4635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.275809 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.275938 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.282105 (+ 6167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:24.282123 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:24.282147 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:24.282203 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:24.286872 (+ 4669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:24.287961 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:24.287973 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:24.287978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:24.288610 (+ 632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:24.289107 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:24.289112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:24.289115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:24.289715 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:24.290093 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:24.290440 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 01:06:24.290717 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 01:06:24.292090 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42852
I20260619 01:06:24.315675 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.296546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.0.97.62:41439)
0619 01:06:24.296780 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:24.296788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:24.297865 (+ 1077us) server_negotiation.cc:407] Connection header received
0619 01:06:24.298066 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:24.298072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:24.298176 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:24.298303 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:24.300774 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.302009 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.304010 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.304733 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.304874 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.308339 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:24.308346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:24.308351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:24.308402 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:24.312358 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:24.313410 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:24.313417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:24.313421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:24.313577 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:24.314017 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:24.314021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:24.314025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:24.314412 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:24.314653 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:24.315074 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 01:06:24.315332 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260619 01:06:24.316521 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47420
I20260619 01:06:24.319103 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 1584
I20260619 01:06:24.339560 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.321469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.0.97.1:38513)
0619 01:06:24.321664 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:24.321673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:24.321696 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:06:24.321770 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:24.321776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:24.321864 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:24.321982 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:24.324118 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.325371 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.327633 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.328304 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.328403 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.331840 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:24.331848 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:24.331853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:24.331896 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:24.335920 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:24.336924 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:24.336931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:24.336935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:24.337114 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:24.337589 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:24.337594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:24.337602 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:24.337982 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:24.338216 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:24.338793 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 01:06:24.339043 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 01:06:24.340901 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42866
W20260619 01:06:24.341514 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 1584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:06:24.341725 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 1584
I20260619 01:06:24.361811 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 1724
Jun 19 01:06:24 dist-test-slave-5p8w krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831183, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:06:24.377602 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.362582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.0.97.2:38405)
0619 01:06:24.362771 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:24.362778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:24.362939 (+ 161us) server_negotiation.cc:407] Connection header received
0619 01:06:24.363152 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:24.363164 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:24.363243 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:24.363350 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:24.364527 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.365252 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.366541 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.366972 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.367110 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.371695 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:24.371712 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:24.371729 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:24.371774 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:24.374751 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:24.375476 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:24.375484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:24.375486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:24.375628 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:24.376002 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:24.376009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:24.376012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:24.376379 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:24.376702 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:24.376993 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 01:06:24.377220 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 01:06:24.378335 1854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58446
W20260619 01:06:24.379014 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 1724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:06:24.379199 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 1724
I20260619 01:06:24.403564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 1863
Jun 19 01:06:24 dist-test-slave-5p8w krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831183, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:06:24.424619 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:24.404548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.0.97.3:45927)
0619 01:06:24.404756 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:24.404762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:24.404783 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:06:24.404884 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:24.404891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:24.404988 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:24.405119 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:24.406777 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.408039 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.410181 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:24.410902 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:24.411034 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:24.416687 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:24.416702 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:24.416723 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:24.416771 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:24.421105 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:24.422076 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:24.422086 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:24.422090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:24.422265 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:24.422705 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:24.422710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:24.422714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:24.423150 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:24.423482 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:24.423803 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 01:06:24.424037 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 01:06:24.425458 1993 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40924
W20260619 01:06:24.426203 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 1863: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:06:24.426362 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 1863
I20260619 01:06:24.448127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 1481
I20260619 01:06:24.449379 1553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:24.565217 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 1481
2026-06-19T01:06:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4688 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:06:24 dist-test-slave-5p8w krb5kdc[2083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:06:24 dist-test-slave-5p8w krb5kdc[2083](info): set up 2 sockets
Jun 19 01:06:24 dist-test-slave-5p8w krb5kdc[2083](info): commencing operation
krb5kdc: starting...
W20260619 01:06:25.657658 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:06:25 dist-test-slave-5p8w krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:06:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:06:25.857604 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39077
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:26.213109 2096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:26.213539 2096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:26.213732 2096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:26.224143 2096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:26.224311 2096 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 01:06:26.224383 2096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:26.224488 2096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:26.224583 2096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:26.224702 2096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:06:26.237125 2096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39077
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:26.238633 2096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:26.240118 2096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:26.253209 2104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:26.253218 2102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:26.253611 2101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:26.254125 2096 server_base.cc:1034] running on GCE node
I20260619 01:06:26.254990 2096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:26.256631 2096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:26.257848 2096 hybrid_clock.cc:648] HybridClock initialized: now 1781831186257783 us; error 42 us; skew 500 ppm
I20260619 01:06:26.258237 2096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:26 dist-test-slave-5p8w krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:26.267450 2096 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:06:26.269282 2096 webserver.cc:460] Webserver started at http://127.0.97.62:34829/ using document root <none> and password file <none>
I20260619 01:06:26.270138 2096 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:26.270277 2096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:26.270628 2096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:26.273926 2096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "c31350d3c3714f82b2d3266cdce9049a"
format_stamp: "Formatted at 2026-06-19 01:06:26 on dist-test-slave-5p8w"
server_key: "3ce4e9f23bad7948110c92a7f4f21660"
server_key_iv: "02562045183e23bdcdf28d105d031303"
server_key_version: "encryptionkey@0"
I20260619 01:06:26.274763 2096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "c31350d3c3714f82b2d3266cdce9049a"
format_stamp: "Formatted at 2026-06-19 01:06:26 on dist-test-slave-5p8w"
server_key: "3ce4e9f23bad7948110c92a7f4f21660"
server_key_iv: "02562045183e23bdcdf28d105d031303"
server_key_version: "encryptionkey@0"
I20260619 01:06:26.280499 2096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:06:26.284387 2111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:26.286006 2096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:26.286217 2096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "c31350d3c3714f82b2d3266cdce9049a"
format_stamp: "Formatted at 2026-06-19 01:06:26 on dist-test-slave-5p8w"
server_key: "3ce4e9f23bad7948110c92a7f4f21660"
server_key_iv: "02562045183e23bdcdf28d105d031303"
server_key_version: "encryptionkey@0"
I20260619 01:06:26.286461 2096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:26.306300 2096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:26.311199 2096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:26.311496 2096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:26.335438 2096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39077
I20260619 01:06:26.335449 2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39077 every 8 connection(s)
I20260619 01:06:26.336920 2096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:26.337903 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 2096
I20260619 01:06:26.338151 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:26.338865 388 external_mini_cluster.cc:1428] Setting key 16cec3d8118753623b26b88dded83c4a
I20260619 01:06:26.342002 2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:06:26 dist-test-slave-5p8w krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:26.355976 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a: Bootstrap starting.
I20260619 01:06:26.360792 2173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:26.362205 2173 log.cc:824] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:26.366010 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a: No bootstrap required, opened a new log
I20260619 01:06:26.366051 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:26.341133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.0.97.62:39077)
0619 01:06:26.341990 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:26.342008 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:26.342063 (+ 55us) server_negotiation.cc:407] Connection header received
0619 01:06:26.344418 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:26.344450 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:26.345000 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:26.345589 (+ 589us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:26.347089 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:26.349349 (+ 2260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:26.350747 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:26.351821 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:26.352027 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:26.356262 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:26.356275 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:26.356300 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:26.356366 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:26.360607 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:26.361655 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:26.361664 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:26.361673 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:26.361903 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:26.362414 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:26.362418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:26.362421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:26.363211 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:26.363439 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:26.364073 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 01:06:26.364563 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":372,"threads_started":1}
I20260619 01:06:26.374662 2173 raft_consensus.cc:348] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31350d3c3714f82b2d3266cdce9049a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39077 } }
I20260619 01:06:26.375089 2173 raft_consensus.cc:374] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:26.375229 2173 raft_consensus.cc:729] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31350d3c3714f82b2d3266cdce9049a, State: Initialized, Role: FOLLOWER
I20260619 01:06:26.375809 2173 consensus_queue.cc:260] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31350d3c3714f82b2d3266cdce9049a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39077 } }
I20260619 01:06:26.376009 2173 raft_consensus.cc:388] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:26.376108 2173 raft_consensus.cc:482] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:26.376267 2173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:26.425665 2173 raft_consensus.cc:504] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31350d3c3714f82b2d3266cdce9049a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39077 } }
I20260619 01:06:26.426412 2173 leader_election.cc:302] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c31350d3c3714f82b2d3266cdce9049a; no voters:
I20260619 01:06:26.427160 2173 leader_election.cc:288] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:26.427359 2178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:26.428691 2178 raft_consensus.cc:686] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [term 1 LEADER]: Becoming Leader. State: Replica: c31350d3c3714f82b2d3266cdce9049a, State: Running, Role: LEADER
I20260619 01:06:26.429517 2178 consensus_queue.cc:237] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31350d3c3714f82b2d3266cdce9049a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39077 } }
I20260619 01:06:26.429728 2173 sys_catalog.cc:564] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:26.434185 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31350d3c3714f82b2d3266cdce9049a. Latest consensus state: current_term: 1 leader_uuid: "c31350d3c3714f82b2d3266cdce9049a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31350d3c3714f82b2d3266cdce9049a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39077 } } }
I20260619 01:06:26.434674 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:26.434257 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c31350d3c3714f82b2d3266cdce9049a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c31350d3c3714f82b2d3266cdce9049a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39077 } } }
I20260619 01:06:26.434886 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:26.436830 2185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:26.442610 2185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:26.453768 2185 catalog_manager.cc:1347] Generated new cluster ID: 3b6d282f5cd443c0aec5fde230fcccf2
I20260619 01:06:26.453936 2185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:26.472105 2185 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:26.473635 2185 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:26.481194 2185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a: Generated new TSK 0
I20260619 01:06:26.482260 2185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
W20260619 01:06:27.494069 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.1: real 0.995s user 0.001s sys 0.009s
I20260619 01:06:27.495162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39077
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:27.835772 2199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:27.836133 2199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:27.836321 2199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:27.843896 2199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:27.844064 2199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:27.844165 2199 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 01:06:27.844219 2199 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:06:27.844288 2199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:06:27.854672 2199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:27.855998 2199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:27.857280 2199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:27.873878 2207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:27.873952 2204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:27.873908 2205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:27.874953 2199 server_base.cc:1034] running on GCE node
I20260619 01:06:27.875646 2199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:27.876632 2199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:27.877875 2199 hybrid_clock.cc:648] HybridClock initialized: now 1781831187877825 us; error 52 us; skew 500 ppm
I20260619 01:06:27.878232 2199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:27 dist-test-slave-5p8w krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:27.881701 2199 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:06:27.883364 2199 webserver.cc:460] Webserver started at http://127.0.97.1:39571/ using document root <none> and password file <none>
I20260619 01:06:27.884122 2199 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:27.884274 2199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:27.884600 2199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:27.887823 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "18074e45ff374a3fb8105f7e6ee200bf"
format_stamp: "Formatted at 2026-06-19 01:06:27 on dist-test-slave-5p8w"
server_key: "70a8f4e6e62c7c47af1094a9980e8389"
server_key_iv: "b7d389fd7eff7047a2d7f5e1ca3ae4a7"
server_key_version: "encryptionkey@0"
I20260619 01:06:27.888676 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "18074e45ff374a3fb8105f7e6ee200bf"
format_stamp: "Formatted at 2026-06-19 01:06:27 on dist-test-slave-5p8w"
server_key: "70a8f4e6e62c7c47af1094a9980e8389"
server_key_iv: "b7d389fd7eff7047a2d7f5e1ca3ae4a7"
server_key_version: "encryptionkey@0"
I20260619 01:06:27.894455 2199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 01:06:27.898388 2214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:27.899884 2199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 01:06:27.900085 2199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "18074e45ff374a3fb8105f7e6ee200bf"
format_stamp: "Formatted at 2026-06-19 01:06:27 on dist-test-slave-5p8w"
server_key: "70a8f4e6e62c7c47af1094a9980e8389"
server_key_iv: "b7d389fd7eff7047a2d7f5e1ca3ae4a7"
server_key_version: "encryptionkey@0"
I20260619 01:06:27.900315 2199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:27.929626 2199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:27.934099 2199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:27.934424 2199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:27.937225 2199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:27.937378 2199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 01:06:27.937532 2199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:27.937630 2199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:06:27 dist-test-slave-5p8w krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:27.965906 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:27.937895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53823 (local address 127.0.97.62:39077)
0619 01:06:27.938523 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:27.938530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:27.939897 (+ 1367us) server_negotiation.cc:407] Connection header received
0619 01:06:27.942539 (+ 2642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:27.942544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:27.942640 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:27.942778 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:27.945398 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:27.946727 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:27.950436 (+ 3709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:27.951238 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:27.951391 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:27.957439 (+ 6048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:27.957450 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:27.957459 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:27.957521 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:27.961752 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:27.962993 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:27.963003 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:27.963006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:27.963182 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:27.963865 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:27.963873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:27.963878 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:27.964232 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:27.964387 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:27.965236 (+ 849us) server_negotiation.cc:299] Negotiation successful
0619 01:06:27.965505 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":396,"threads_started":1}
I20260619 01:06:27.967005 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:27.938684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39077 (local address 127.0.97.1:53823)
0619 01:06:27.939675 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:27.939729 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:27.942096 (+ 2367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:27.943145 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:27.943159 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:27.944244 (+ 1085us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:27.945116 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:27.945138 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:27.947033 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:27.947042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:27.950105 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:27.950140 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:27.951789 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:27.951797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:27.952265 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:27.953502 (+ 1237us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:27.953531 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:27.957007 (+ 3476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:27.961967 (+ 4960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:27.961985 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:27.962012 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:27.962817 (+ 805us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:27.963362 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:27.963370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:27.963379 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:27.963667 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:27.964455 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:27.964462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:27.964842 (+ 380us) client_negotiation.cc:769] Sending connection context
0619 01:06:27.965125 (+ 283us) client_negotiation.cc:240] Negotiation successful
0619 01:06:27.965781 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":414,"threads_started":1}
I20260619 01:06:27.988451 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:27.973663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50821 (local address 127.0.97.62:39077)
0619 01:06:27.973847 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:27.973852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:27.974566 (+ 714us) server_negotiation.cc:407] Connection header received
0619 01:06:27.974751 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:27.974760 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:27.974872 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:27.975008 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:27.975910 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:27.977144 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:27.979185 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:27.979893 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:27.980072 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:27.982350 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:27.982355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:27.982359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:27.982393 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:27.985174 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:27.986133 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:27.986138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:27.986140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:27.986265 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:27.986660 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:27.986665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:27.986668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:27.986937 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:27.987111 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:27.987749 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 01:06:27.988029 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:06:27.989040 2303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:27.973673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39077 (local address 127.0.97.1:50821)
0619 01:06:27.974376 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:27.974445 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:27.974611 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:27.975046 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:27.975049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:27.975478 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:27.975762 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:27.975771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:27.977355 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:27.977363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:27.979008 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:27.979029 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:27.980104 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:27.980109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:27.980236 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:27.980979 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:27.980997 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:27.982128 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:27.985361 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:27.985367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:27.985371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:27.985950 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:27.986383 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:27.986391 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:27.986399 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:27.986524 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:27.987079 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:27.987088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:27.987385 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 01:06:27.987557 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 01:06:27.988064 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":411,"threads_started":1}
I20260619 01:06:27.995440 2199 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38115
I20260619 01:06:27.995486 2334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38115 every 8 connection(s)
I20260619 01:06:27.996765 2199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:06:27.998950 2335 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39077
I20260619 01:06:27.999219 2335 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:27.999913 2335 heartbeater.cc:511] Master 127.0.97.62:39077 requested a full tablet report, sending...
I20260619 01:06:28.002192 2128 ts_manager.cc:194] Registered new tserver with Master: 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1:38115)
I20260619 01:06:28.003813 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53823
I20260619 01:06:28.005445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 2199
I20260619 01:06:28.005707 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:06:28.006323 388 external_mini_cluster.cc:1428] Setting key 5a82decccc06566d853abe83b224a9a3
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:06:28.049101 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39077
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:28.400130 2339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:28.400529 2339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:28.400739 2339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:28.408597 2339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:28.408746 2339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:28.408931 2339 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 01:06:28.409025 2339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:06:28.409240 2339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:06:28.419700 2339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:28.421201 2339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:28.422747 2339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:28.437750 2345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:28.437767 2344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:28.437953 2347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:28.439183 2339 server_base.cc:1034] running on GCE node
I20260619 01:06:28.439916 2339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:28.441088 2339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:28.442373 2339 hybrid_clock.cc:648] HybridClock initialized: now 1781831188442298 us; error 68 us; skew 500 ppm
I20260619 01:06:28.442771 2339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:28 dist-test-slave-5p8w krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:28.446281 2339 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:06:28.447975 2339 webserver.cc:460] Webserver started at http://127.0.97.2:45449/ using document root <none> and password file <none>
I20260619 01:06:28.448705 2339 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:28.448820 2339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:28.449123 2339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:28.452570 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9"
format_stamp: "Formatted at 2026-06-19 01:06:28 on dist-test-slave-5p8w"
server_key: "688b1d33ffd706fc9f4ed37113d30a29"
server_key_iv: "f9ac9baeb3fd941f2d7627749f123e30"
server_key_version: "encryptionkey@0"
I20260619 01:06:28.453579 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9"
format_stamp: "Formatted at 2026-06-19 01:06:28 on dist-test-slave-5p8w"
server_key: "688b1d33ffd706fc9f4ed37113d30a29"
server_key_iv: "f9ac9baeb3fd941f2d7627749f123e30"
server_key_version: "encryptionkey@0"
I20260619 01:06:28.459517 2339 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 01:06:28.463570 2354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:28.465145 2339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 01:06:28.465396 2339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9"
format_stamp: "Formatted at 2026-06-19 01:06:28 on dist-test-slave-5p8w"
server_key: "688b1d33ffd706fc9f4ed37113d30a29"
server_key_iv: "f9ac9baeb3fd941f2d7627749f123e30"
server_key_version: "encryptionkey@0"
I20260619 01:06:28.465610 2339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:28.479216 2339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:28.483421 2339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:28.483692 2339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:28.486625 2339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:28.486730 2339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:28.486871 2339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:28.487779 2339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:06:28 dist-test-slave-5p8w krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:28.513897 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:28.487088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51071 (local address 127.0.97.62:39077)
0619 01:06:28.487275 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:28.487280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:28.489261 (+ 1981us) server_negotiation.cc:407] Connection header received
0619 01:06:28.491778 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:28.491785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:28.491884 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:28.492025 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:28.494840 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:28.496251 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:28.499588 (+ 3337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:28.500407 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:28.500587 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:28.505900 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:28.505906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:28.505909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:28.505947 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:28.509432 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:28.510714 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:28.510719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:28.510722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:28.510850 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:28.511460 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:28.511467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:28.511472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:28.511959 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:28.512153 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:28.513221 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0619 01:06:28.513522 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 01:06:28.515591 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:28.487704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39077 (local address 127.0.97.2:51071)
0619 01:06:28.489030 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:28.489086 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:28.491291 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:28.492386 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:28.492397 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:28.493400 (+ 1003us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:28.494559 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:28.494598 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:28.496543 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:28.496551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:28.499318 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:28.499349 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:28.500762 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:28.500774 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:28.501130 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:28.502114 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:28.502159 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:28.505582 (+ 3423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:28.509623 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:28.509634 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:28.509660 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:28.510489 (+ 829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:28.510986 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:28.510991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:28.510995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:28.511268 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:28.512126 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:28.512137 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:28.512688 (+ 551us) client_negotiation.cc:769] Sending connection context
0619 01:06:28.513120 (+ 432us) client_negotiation.cc:240] Negotiation successful
0619 01:06:28.513816 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"thread_start_us":842,"threads_started":1}
I20260619 01:06:28.535156 2339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40229
I20260619 01:06:28.535336 2473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40229 every 8 connection(s)
I20260619 01:06:28.536559 2339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:06:28.538952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 2339
I20260619 01:06:28.539158 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:06:28.539237 2474 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39077
I20260619 01:06:28.539445 2474 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:28.539709 388 external_mini_cluster.cc:1428] Setting key 42a13719d5fd2cd6b564f95b39f92003
I20260619 01:06:28.540108 2474 heartbeater.cc:511] Master 127.0.97.62:39077 requested a full tablet report, sending...
I20260619 01:06:28.540881 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:28.523547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39077 (local address 127.0.97.2:53429)
0619 01:06:28.524362 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:28.524384 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:28.524603 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:28.525167 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:28.525172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:28.525896 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:28.526473 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:28.526485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:28.527890 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:28.527899 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:28.530220 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:28.530243 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:28.531380 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:28.531390 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:28.531644 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:28.532424 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:28.532457 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:28.533681 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:28.537594 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:28.537601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:28.537606 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:28.538152 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:28.538727 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:28.538733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:28.538737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:28.538891 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:28.539637 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:28.539642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:28.539880 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 01:06:28.540022 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 01:06:28.540397 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":477,"threads_started":1}
I20260619 01:06:28.541846 2128 ts_manager.cc:194] Registered new tserver with Master: 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2:40229)
I20260619 01:06:28.541962 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:28.523553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53429 (local address 127.0.97.62:39077)
0619 01:06:28.523775 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:28.523783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:28.524502 (+ 719us) server_negotiation.cc:407] Connection header received
0619 01:06:28.524836 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:28.524844 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:28.524947 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:28.525078 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:28.526637 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:28.527694 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:28.530418 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:28.531119 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:28.531291 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:28.533890 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:28.533899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:28.533906 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:28.533953 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:28.537373 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:28.538365 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:28.538370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:28.538373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:28.538573 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:28.539004 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:28.539009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:28.539013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:28.539488 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:28.541140 (+ 1652us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:28.541332 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 01:06:28.541601 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:06:28.543047 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51071
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:06:28.585968 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39077
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:28.930387 2478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:28.930754 2478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:28.930970 2478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:28.938773 2478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:28.938917 2478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:28.939076 2478 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 01:06:28.939137 2478 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:06:28.939234 2478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:06:28.949652 2478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:28.951114 2478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:28.952862 2478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:28.967455 2484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:28.967495 2483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:28.967566 2486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:28.968940 2478 server_base.cc:1034] running on GCE node
I20260619 01:06:28.969750 2478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:28.970912 2478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:28.972133 2478 hybrid_clock.cc:648] HybridClock initialized: now 1781831188972085 us; error 44 us; skew 500 ppm
I20260619 01:06:28.972488 2478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:28 dist-test-slave-5p8w krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:28.976034 2478 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:06:28.977692 2478 webserver.cc:460] Webserver started at http://127.0.97.3:34681/ using document root <none> and password file <none>
I20260619 01:06:28.978433 2478 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:28.978542 2478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:28.978817 2478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:28.982103 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "318291c09cef44cab20a45bf482185cb"
format_stamp: "Formatted at 2026-06-19 01:06:28 on dist-test-slave-5p8w"
server_key: "738f5c5148ed6ce6f83b89c8b709550b"
server_key_iv: "1e95cf6d2809505a84653a120c62f449"
server_key_version: "encryptionkey@0"
I20260619 01:06:28.982940 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "318291c09cef44cab20a45bf482185cb"
format_stamp: "Formatted at 2026-06-19 01:06:28 on dist-test-slave-5p8w"
server_key: "738f5c5148ed6ce6f83b89c8b709550b"
server_key_iv: "1e95cf6d2809505a84653a120c62f449"
server_key_version: "encryptionkey@0"
I20260619 01:06:28.988744 2478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:06:28.992947 2493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:28.994659 2478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:28.994858 2478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "318291c09cef44cab20a45bf482185cb"
format_stamp: "Formatted at 2026-06-19 01:06:28 on dist-test-slave-5p8w"
server_key: "738f5c5148ed6ce6f83b89c8b709550b"
server_key_iv: "1e95cf6d2809505a84653a120c62f449"
server_key_version: "encryptionkey@0"
I20260619 01:06:28.995077 2478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:29.007366 2335 heartbeater.cc:503] Master 127.0.97.62:39077 was elected leader, sending a full tablet report...
I20260619 01:06:29.012632 2478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:29.017575 2478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:29.017953 2478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:29.021070 2478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:29.021224 2478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:29.021391 2478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:29.021494 2478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:06:29 dist-test-slave-5p8w krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:29.048741 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.022071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44443 (local address 127.0.97.62:39077)
0619 01:06:29.022298 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.022307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.023974 (+ 1667us) server_negotiation.cc:407] Connection header received
0619 01:06:29.026652 (+ 2678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.026674 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.026806 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.026953 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:29.029884 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.031078 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.033671 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.034333 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.034490 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.040333 (+ 5843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:29.040345 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:29.040353 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:29.040411 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:29.044453 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:29.045824 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:29.045830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:29.045832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:29.045969 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:29.046641 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:29.046647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:29.046650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:29.046985 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:29.047200 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.048095 (+ 895us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.048342 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 01:06:29.050185 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.022714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39077 (local address 127.0.97.3:44443)
0619 01:06:29.023721 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.023780 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.026202 (+ 2422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.027323 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.027334 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.028356 (+ 1022us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:29.029602 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.029649 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.031275 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.031281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.033472 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.033494 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.034623 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.034631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.034915 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.036061 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:29.036104 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:29.039994 (+ 3890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:29.044736 (+ 4742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:29.044753 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:29.044783 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:29.045657 (+ 874us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:29.046159 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:29.046167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:29.046172 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:29.046477 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:29.047130 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:29.047139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:29.047680 (+ 541us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.047989 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.048579 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":430,"threads_started":1}
I20260619 01:06:29.074138 2478 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43637
I20260619 01:06:29.074187 2612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43637 every 8 connection(s)
I20260619 01:06:29.074970 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.057782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39077 (local address 127.0.97.3:46261)
0619 01:06:29.058975 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.059005 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.059219 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.059683 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.059689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.060409 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:29.060872 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.060895 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.062389 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.062402 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.063769 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.063784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.064748 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.064755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.065011 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.065904 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:29.065938 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:29.067385 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:29.072210 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:29.072214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:29.072217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:29.072630 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:29.073192 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:29.073195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:29.073197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:29.073385 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:29.073935 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:29.073940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:29.074174 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.074341 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.074652 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"thread_start_us":753,"threads_started":1}
I20260619 01:06:29.075598 2478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:06:29.075796 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.057775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46261 (local address 127.0.97.62:39077)
0619 01:06:29.058269 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.058277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.059140 (+ 863us) server_negotiation.cc:407] Connection header received
0619 01:06:29.059406 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.059413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.059523 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.059642 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:29.061115 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.062228 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.063949 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.064534 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.064673 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.067700 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:29.067711 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:29.067716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:29.067758 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:29.072068 (+ 4310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:29.072827 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:29.072832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:29.072836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:29.073023 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:29.073510 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:29.073518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:29.073522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:29.073814 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:29.073993 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.075321 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.075513 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332}
I20260619 01:06:29.076277 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 2478
I20260619 01:06:29.076508 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:06:29.077315 388 external_mini_cluster.cc:1428] Setting key 59a5767b62c746ccd211a3e29d237f21
I20260619 01:06:29.077872 2613 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39077
I20260619 01:06:29.078285 2613 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:29.079210 2613 heartbeater.cc:511] Master 127.0.97.62:39077 requested a full tablet report, sending...
I20260619 01:06:29.080869 2127 ts_manager.cc:194] Registered new tserver with Master: 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637)
I20260619 01:06:29.082253 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:06:29.082288 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44443
Jun 19 01:06:29 dist-test-slave-5p8w krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 01:06:29 dist-test-slave-5p8w krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:29.131831 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.117214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.0.97.62:39077)
0619 01:06:29.117483 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.117489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.117655 (+ 166us) server_negotiation.cc:407] Connection header received
0619 01:06:29.117811 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.117816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.117889 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.117984 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:29.119428 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.120174 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.121442 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.121858 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.121957 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.125957 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:29.125962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:29.125965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:29.126005 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:29.128436 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:29.129116 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:29.129121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:29.129124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:29.129356 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:29.129899 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:29.129907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:29.129913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:29.130334 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:29.130596 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.131044 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.131327 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260619 01:06:29.137348 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:06:29.146214 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:06:29.173362 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.159681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.0.97.3:43637)
0619 01:06:29.160898 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.160908 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.160956 (+ 48us) server_negotiation.cc:407] Connection header received
0619 01:06:29.161808 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.161817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.162125 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.162314 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:29.163738 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.166068 (+ 2330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.169787 (+ 3719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.171469 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.171610 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.171883 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.172725 (+ 842us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.172961 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":792,"threads_started":1}
I20260619 01:06:29.173456 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.159250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43637 (local address 127.0.0.1:56736)
0619 01:06:29.160341 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.160401 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.161467 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.162720 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.162733 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.162777 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:29.163405 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.163422 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.166313 (+ 2891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.166323 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.169584 (+ 3261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.169602 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.172046 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.172054 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.172332 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.172423 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.172580 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.172921 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":60,"thread_start_us":644,"threads_started":1}
I20260619 01:06:29.176730 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.162033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38115 (local address 127.0.0.1:40150)
0619 01:06:29.168801 (+ 6768us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.168826 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.168947 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.169804 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.169809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.169836 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:29.170321 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.170336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.171952 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.171963 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.174331 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.174346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.175852 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.175862 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.176038 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.176080 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.176196 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.176340 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6617,"mutex_wait_us":134,"thread_start_us":6456,"threads_started":1}
I20260619 01:06:29.177603 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.162284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.0.97.1:38115)
0619 01:06:29.169099 (+ 6815us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.169123 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.169148 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:06:29.169272 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.169279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.169600 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.169823 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:29.170548 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.171798 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.174505 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.175626 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.175767 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.175995 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.176265 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.176452 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6643,"thread_start_us":6516,"threads_started":1}
I20260619 01:06:29.178259 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.160871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40229 (local address 127.0.0.1:56230)
0619 01:06:29.161848 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.161867 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.168190 (+ 6323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.169999 (+ 1809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.170004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.170023 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:29.170543 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.170554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.172248 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.172254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.175246 (+ 2992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.175263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.177383 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.177389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.177579 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.177611 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.177711 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.177856 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":23,"thread_start_us":727,"threads_started":1}
I20260619 01:06:29.178300 2543 tablet_service.cc:1467] Processing CreateTablet for tablet c340a84f1808446bbeae7f42df42e19d (DEFAULT_TABLE table=test-table [id=3331479ca7114183bafc60eb037c2394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:29.179085 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.161101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.0.97.2:40229)
0619 01:06:29.167785 (+ 6684us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.167796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.167832 (+ 36us) server_negotiation.cc:407] Connection header received
0619 01:06:29.169150 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.169171 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.169540 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.169759 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:29.170702 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.172049 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.175452 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.176876 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.177009 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.177212 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.178612 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.178821 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6474,"thread_start_us":6951,"threads_started":1}
I20260619 01:06:29.180017 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c340a84f1808446bbeae7f42df42e19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:29.181982 2404 tablet_service.cc:1467] Processing CreateTablet for tablet c340a84f1808446bbeae7f42df42e19d (DEFAULT_TABLE table=test-table [id=3331479ca7114183bafc60eb037c2394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:29.181981 2265 tablet_service.cc:1467] Processing CreateTablet for tablet c340a84f1808446bbeae7f42df42e19d (DEFAULT_TABLE table=test-table [id=3331479ca7114183bafc60eb037c2394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:29.183367 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c340a84f1808446bbeae7f42df42e19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:29.183399 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c340a84f1808446bbeae7f42df42e19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:29.195343 2632 tablet_bootstrap.cc:492] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Bootstrap starting.
I20260619 01:06:29.198092 2633 tablet_bootstrap.cc:492] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: Bootstrap starting.
I20260619 01:06:29.199419 2632 tablet_bootstrap.cc:654] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:29.201438 2632 log.cc:824] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:29.201891 2633 tablet_bootstrap.cc:654] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:29.203291 2633 log.cc:824] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:29.204978 2632 tablet_bootstrap.cc:492] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: No bootstrap required, opened a new log
I20260619 01:06:29.205320 2632 ts_tablet_manager.cc:1397] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 01:06:29.205415 2634 tablet_bootstrap.cc:492] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: Bootstrap starting.
I20260619 01:06:29.206525 2633 tablet_bootstrap.cc:492] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: No bootstrap required, opened a new log
I20260619 01:06:29.206820 2633 ts_tablet_manager.cc:1397] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 01:06:29.208500 2634 tablet_bootstrap.cc:654] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:29.209707 2634 log.cc:824] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:29.213326 2634 tablet_bootstrap.cc:492] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: No bootstrap required, opened a new log
I20260619 01:06:29.213672 2634 ts_tablet_manager.cc:1397] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260619 01:06:29.216560 2632 raft_consensus.cc:348] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } }
I20260619 01:06:29.217082 2632 raft_consensus.cc:374] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:29.217196 2632 raft_consensus.cc:729] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73602c8a32ab4f1ca995e6c5e1d1d9e9, State: Initialized, Role: FOLLOWER
I20260619 01:06:29.217811 2633 raft_consensus.cc:348] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } }
I20260619 01:06:29.218003 2632 consensus_queue.cc:260] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } }
I20260619 01:06:29.218272 2633 raft_consensus.cc:374] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:29.218369 2633 raft_consensus.cc:729] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318291c09cef44cab20a45bf482185cb, State: Initialized, Role: FOLLOWER
I20260619 01:06:29.219206 2633 consensus_queue.cc:260] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } }
I20260619 01:06:29.220538 2613 heartbeater.cc:503] Master 127.0.97.62:39077 was elected leader, sending a full tablet report...
I20260619 01:06:29.221753 2633 ts_tablet_manager.cc:1428] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260619 01:06:29.222710 2634 raft_consensus.cc:348] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } }
I20260619 01:06:29.223186 2634 raft_consensus.cc:374] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:29.223294 2634 raft_consensus.cc:729] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18074e45ff374a3fb8105f7e6ee200bf, State: Initialized, Role: FOLLOWER
I20260619 01:06:29.223958 2634 consensus_queue.cc:260] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } }
I20260619 01:06:29.226106 2632 ts_tablet_manager.cc:1428] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20260619 01:06:29.226545 2474 heartbeater.cc:503] Master 127.0.97.62:39077 was elected leader, sending a full tablet report...
I20260619 01:06:29.228006 2634 ts_tablet_manager.cc:1428] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
W20260619 01:06:29.249423 2336 tablet.cc:2369] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:06:29.287479 2638 raft_consensus.cc:482] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:29.287914 2638 raft_consensus.cc:504] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } }
W20260619 01:06:29.288738 2475 tablet.cc:2369] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:06:29.289794 2638 leader_election.cc:288] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637), 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1:38115)
I20260619 01:06:29.296980 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.290263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59117 (local address 127.0.97.3:43637)
0619 01:06:29.290477 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.290483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.291093 (+ 610us) server_negotiation.cc:407] Connection header received
0619 01:06:29.291196 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.291201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.291278 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.291368 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:29.292160 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.293184 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.295085 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.295855 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.296030 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.296144 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.296559 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.296687 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 01:06:29.297076 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.291077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40779 (local address 127.0.97.1:38115)
0619 01:06:29.291301 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.291308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.291740 (+ 432us) server_negotiation.cc:407] Connection header received
0619 01:06:29.291974 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.291981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.292065 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.292180 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:29.292649 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.293559 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.295270 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.296055 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.296223 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.296329 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.296700 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.296827 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 01:06:29.297189 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.290050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43637 (local address 127.0.97.2:59117)
0619 01:06:29.290750 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.290769 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.290916 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.291616 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.291620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.291642 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:29.292008 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.292017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.293396 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.293400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.294901 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.294913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.296093 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.296100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.296294 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.296379 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.296534 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.296679 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":47,"thread_start_us":329,"threads_started":1}
I20260619 01:06:29.297238 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.290803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38115 (local address 127.0.97.2:40779)
0619 01:06:29.291608 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.291630 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.291818 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.292196 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.292199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.292218 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:29.292463 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.292470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.293698 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.293704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.295107 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.295126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.296288 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.296294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.296542 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.296564 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.296676 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.296806 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":1,"thread_start_us":369,"threads_started":1}
I20260619 01:06:29.298368 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c340a84f1808446bbeae7f42df42e19d" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318291c09cef44cab20a45bf482185cb" is_pre_election: true
I20260619 01:06:29.298368 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c340a84f1808446bbeae7f42df42e19d" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18074e45ff374a3fb8105f7e6ee200bf" is_pre_election: true
I20260619 01:06:29.299000 2285 raft_consensus.cc:2454] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 0.
I20260619 01:06:29.299000 2563 raft_consensus.cc:2454] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 0.
I20260619 01:06:29.299793 2357 leader_election.cc:302] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18074e45ff374a3fb8105f7e6ee200bf, 73602c8a32ab4f1ca995e6c5e1d1d9e9; no voters:
I20260619 01:06:29.300313 2638 raft_consensus.cc:2781] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:06:29.300495 2638 raft_consensus.cc:482] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:06:29.300623 2638 raft_consensus.cc:3037] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:29.303110 2638 raft_consensus.cc:504] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } }
I20260619 01:06:29.303890 2638 leader_election.cc:288] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 election: Requested vote from peers 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637), 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1:38115)
I20260619 01:06:29.304553 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c340a84f1808446bbeae7f42df42e19d" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318291c09cef44cab20a45bf482185cb"
I20260619 01:06:29.304668 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c340a84f1808446bbeae7f42df42e19d" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18074e45ff374a3fb8105f7e6ee200bf"
I20260619 01:06:29.304798 2563 raft_consensus.cc:3037] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:29.304955 2285 raft_consensus.cc:3037] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:29.307296 2563 raft_consensus.cc:2454] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 1.
I20260619 01:06:29.307540 2285 raft_consensus.cc:2454] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 1.
I20260619 01:06:29.307859 2357 leader_election.cc:302] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 318291c09cef44cab20a45bf482185cb, 73602c8a32ab4f1ca995e6c5e1d1d9e9; no voters:
I20260619 01:06:29.308246 2638 raft_consensus.cc:2781] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:29.308734 2638 raft_consensus.cc:686] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 LEADER]: Becoming Leader. State: Replica: 73602c8a32ab4f1ca995e6c5e1d1d9e9, State: Running, Role: LEADER
I20260619 01:06:29.309332 2638 consensus_queue.cc:237] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } }
I20260619 01:06:29.314402 2127 catalog_manager.cc:5495] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } health_report { overall_health: UNKNOWN } } }
W20260619 01:06:29.328382 2614 tablet.cc:2369] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 01:06:29 dist-test-slave-5p8w krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:29.393486 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.380079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39077 (local address 127.0.0.1:47212)
0619 01:06:29.380755 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.380771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.380898 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.381186 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.381189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.381965 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:29.382229 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.382238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.383165 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.383168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.384136 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.384153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.385348 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.385355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.385487 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.386104 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:29.386120 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:29.388244 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:29.391228 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:29.391240 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:29.391247 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:29.391702 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:29.392019 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:29.392023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:29.392025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:29.392164 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:29.392612 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:29.392619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:29.392775 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.392907 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.393119 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":373,"threads_started":1}
I20260619 01:06:29.393563 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.380042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.0.97.62:39077)
0619 01:06:29.380270 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.380279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.380858 (+ 579us) server_negotiation.cc:407] Connection header received
0619 01:06:29.380993 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.380997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.381061 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.381142 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:29.382352 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.383052 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.384325 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.385095 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.385223 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.388420 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:29.388425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:29.388428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:29.388457 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:29.391094 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:29.391806 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:29.391810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:29.391813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:29.391924 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:29.392252 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:29.392255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:29.392257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:29.392509 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:29.392613 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.392986 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.393249 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":81}
I20260619 01:06:29.398797 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47212:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 01:06:29.399854 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:06:29.406915 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 086a7d1408d646a18b9fe392e783eeac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=210a3d74bd62458b85483e2764a5c47f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:06:29.406904 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 086a7d1408d646a18b9fe392e783eeac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=210a3d74bd62458b85483e2764a5c47f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:06:29.406953 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 086a7d1408d646a18b9fe392e783eeac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=210a3d74bd62458b85483e2764a5c47f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:06:29.407878 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086a7d1408d646a18b9fe392e783eeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:29.407879 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086a7d1408d646a18b9fe392e783eeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:29.407882 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086a7d1408d646a18b9fe392e783eeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:29.414484 2632 tablet_bootstrap.cc:492] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Bootstrap starting.
I20260619 01:06:29.415498 2634 tablet_bootstrap.cc:492] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: Bootstrap starting.
I20260619 01:06:29.415555 2633 tablet_bootstrap.cc:492] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb: Bootstrap starting.
I20260619 01:06:29.416440 2632 tablet_bootstrap.cc:654] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:29.416873 2634 tablet_bootstrap.cc:654] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:29.417259 2633 tablet_bootstrap.cc:654] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:29.419080 2632 tablet_bootstrap.cc:492] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9: No bootstrap required, opened a new log
I20260619 01:06:29.419275 2632 ts_tablet_manager.cc:1397] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260619 01:06:29.419741 2634 tablet_bootstrap.cc:492] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: No bootstrap required, opened a new log
I20260619 01:06:29.419894 2634 ts_tablet_manager.cc:1397] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 01:06:29.420125 2632 raft_consensus.cc:348] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:29.420403 2632 raft_consensus.cc:374] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:29.420532 2633 tablet_bootstrap.cc:492] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb: No bootstrap required, opened a new log
I20260619 01:06:29.420509 2632 raft_consensus.cc:729] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73602c8a32ab4f1ca995e6c5e1d1d9e9, State: Initialized, Role: FOLLOWER
I20260619 01:06:29.420688 2633 ts_tablet_manager.cc:1397] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260619 01:06:29.420640 2634 raft_consensus.cc:348] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:29.420756 2632 consensus_queue.cc:260] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:29.420928 2634 raft_consensus.cc:374] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:29.421026 2634 raft_consensus.cc:729] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18074e45ff374a3fb8105f7e6ee200bf, State: Initialized, Role: FOLLOWER
I20260619 01:06:29.421248 2634 consensus_queue.cc:260] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:29.421423 2632 ts_tablet_manager.cc:1428] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:06:29.421499 2633 raft_consensus.cc:348] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:29.421567 2638 tablet_replica.cc:439] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } }
I20260619 01:06:29.421769 2633 raft_consensus.cc:374] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:29.421859 2638 tablet_replica.cc:442] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:29.421868 2633 raft_consensus.cc:729] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318291c09cef44cab20a45bf482185cb, State: Initialized, Role: FOLLOWER
I20260619 01:06:29.421948 2634 ts_tablet_manager.cc:1428] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:06:29.421782 2640 tablet_replica.cc:439] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } }
I20260619 01:06:29.422207 2640 tablet_replica.cc:442] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:29.422463 2633 consensus_queue.cc:260] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:29.423127 2633 ts_tablet_manager.cc:1428] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:06:29.423009 2639 tablet_replica.cc:439] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } }
I20260619 01:06:29.423400 2639 tablet_replica.cc:442] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:29.719907 2640 raft_consensus.cc:482] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:29.720144 2640 raft_consensus.cc:504] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:29.721661 2640 leader_election.cc:288] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2:40229), 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637)
I20260619 01:06:29.726958 2638 consensus_queue.cc:1035] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:29.730377 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.722938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47879 (local address 127.0.97.3:43637)
0619 01:06:29.723125 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.723131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.723546 (+ 415us) server_negotiation.cc:407] Connection header received
0619 01:06:29.723706 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.723710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.723765 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.723836 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:29.724364 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.725436 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.728042 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.729204 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.729367 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.729460 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.729949 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.730091 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 01:06:29.731228 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.722758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43637 (local address 127.0.97.1:47879)
0619 01:06:29.723431 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.723448 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.723582 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.723875 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.723881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.723910 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:29.724215 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.724222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.725597 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.725603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.727900 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.727918 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.729531 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.729539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.729728 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.729764 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.730669 (+ 905us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.730866 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":29,"thread_start_us":342,"threads_started":1}
I20260619 01:06:29.731848 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a7d1408d646a18b9fe392e783eeac" candidate_uuid: "18074e45ff374a3fb8105f7e6ee200bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318291c09cef44cab20a45bf482185cb" is_pre_election: true
I20260619 01:06:29.732103 2562 raft_consensus.cc:2454] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18074e45ff374a3fb8105f7e6ee200bf in term 0.
I20260619 01:06:29.732621 2217 leader_election.cc:302] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18074e45ff374a3fb8105f7e6ee200bf, 318291c09cef44cab20a45bf482185cb; no voters:
I20260619 01:06:29.733016 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.722099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39601 (local address 127.0.97.2:40229)
0619 01:06:29.723069 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.723075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.723095 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:06:29.723171 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.723181 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.723241 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.723365 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:29.724021 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.725126 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.729421 (+ 4295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.731942 (+ 2521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.732111 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.732222 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.732545 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.732681 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":749,"threads_started":1}
I20260619 01:06:29.733084 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.721985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40229 (local address 127.0.97.1:39601)
0619 01:06:29.722455 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.722472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.722606 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.723407 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.723412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.723488 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:29.723853 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.723864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.725333 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.725339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.727900 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.727918 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.732169 (+ 4251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.732175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.732381 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.732414 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.732516 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.732661 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":146,"thread_start_us":251,"threads_started":1}
I20260619 01:06:29.733114 2640 raft_consensus.cc:2781] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:06:29.733242 2640 raft_consensus.cc:482] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:06:29.733385 2640 raft_consensus.cc:3037] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:29.736304 2640 raft_consensus.cc:504] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:29.737007 2640 leader_election.cc:288] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [CANDIDATE]: Term 1 election: Requested vote from peers 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2:40229), 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637)
I20260619 01:06:29.738256 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a7d1408d646a18b9fe392e783eeac" candidate_uuid: "18074e45ff374a3fb8105f7e6ee200bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318291c09cef44cab20a45bf482185cb"
I20260619 01:06:29.738492 2563 raft_consensus.cc:3037] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:29.738330 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a7d1408d646a18b9fe392e783eeac" candidate_uuid: "18074e45ff374a3fb8105f7e6ee200bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" is_pre_election: true
I20260619 01:06:29.738669 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a7d1408d646a18b9fe392e783eeac" candidate_uuid: "18074e45ff374a3fb8105f7e6ee200bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9"
I20260619 01:06:29.738730 2424 raft_consensus.cc:2454] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18074e45ff374a3fb8105f7e6ee200bf in term 0.
I20260619 01:06:29.739441 2643 consensus_queue.cc:1035] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:29.741367 2563 raft_consensus.cc:2454] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18074e45ff374a3fb8105f7e6ee200bf in term 1.
I20260619 01:06:29.742183 2217 leader_election.cc:302] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18074e45ff374a3fb8105f7e6ee200bf, 318291c09cef44cab20a45bf482185cb; no voters: 73602c8a32ab4f1ca995e6c5e1d1d9e9
I20260619 01:06:29.742518 2640 raft_consensus.cc:2781] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:29.743086 2640 raft_consensus.cc:686] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [term 1 LEADER]: Becoming Leader. State: Replica: 18074e45ff374a3fb8105f7e6ee200bf, State: Running, Role: LEADER
I20260619 01:06:29.743798 2640 consensus_queue.cc:237] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:29.746012 2661 tablet_replica.cc:439] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: TxnStatusTablet state changed. Reason: New leader 18074e45ff374a3fb8105f7e6ee200bf. Latest consensus state: current_term: 1 leader_uuid: "18074e45ff374a3fb8105f7e6ee200bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } }
I20260619 01:06:29.746387 2661 tablet_replica.cc:442] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:06:29.747010 2665 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 01:06:29.747270 2665 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 01:06:29.749969 2126 catalog_manager.cc:5495] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1). New cstate: current_term: 1 leader_uuid: "18074e45ff374a3fb8105f7e6ee200bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } health_report { overall_health: UNKNOWN } } }
I20260619 01:06:29.796152 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.789578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.0.97.1:38115)
0619 01:06:29.789831 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.789836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.789854 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:06:29.790193 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.790198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.790325 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.790498 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:29.791003 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.791781 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.793725 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.794157 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.794262 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.794842 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:29.795513 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:29.795626 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.795782 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.795863 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 01:06:29.796257 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.789466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38115 (local address 127.0.0.1:40164)
0619 01:06:29.789742 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.789788 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.789933 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.790541 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.790547 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.790561 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:06:29.790763 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.790774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.791914 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.791920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.793580 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.793596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.794327 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.794334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.794556 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.794647 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:06:29.795629 (+ 982us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:06:29.795656 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.795753 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.795934 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 01:06:29.804862 2423 raft_consensus.cc:3037] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:29.804937 2563 raft_consensus.cc:1261] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Refusing update from remote peer 18074e45ff374a3fb8105f7e6ee200bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:29.805783 2661 consensus_queue.cc:1035] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:29.808455 2423 raft_consensus.cc:1261] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Refusing update from remote peer 18074e45ff374a3fb8105f7e6ee200bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:29.809365 2661 consensus_queue.cc:1035] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:29.813450 2639 tablet_replica.cc:439] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb: TxnStatusTablet state changed. Reason: New leader 18074e45ff374a3fb8105f7e6ee200bf. Latest consensus state: current_term: 1 leader_uuid: "18074e45ff374a3fb8105f7e6ee200bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } }
I20260619 01:06:29.814477 2639 tablet_replica.cc:442] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:29.816406 2662 mvcc.cc:205] Tried to move back new op lower bound from 7298380553431216128 to 7298380553195302912. Current Snapshot: MvccSnapshot[applied={T|T < 7298380553431216128}]
I20260619 01:06:29.822432 2661 tablet_replica.cc:439] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18074e45ff374a3fb8105f7e6ee200bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } }
I20260619 01:06:29.822715 2661 tablet_replica.cc:442] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:06:29.825927 2639 tablet_replica.cc:439] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18074e45ff374a3fb8105f7e6ee200bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } }
I20260619 01:06:29.826259 2639 tablet_replica.cc:442] T 086a7d1408d646a18b9fe392e783eeac P 318291c09cef44cab20a45bf482185cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:29.831842 2643 tablet_replica.cc:439] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9: TxnStatusTablet state changed. Reason: New leader 18074e45ff374a3fb8105f7e6ee200bf. Latest consensus state: current_term: 1 leader_uuid: "18074e45ff374a3fb8105f7e6ee200bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } }
I20260619 01:06:29.832154 2643 tablet_replica.cc:442] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:29.832266 2661 tablet_replica.cc:439] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18074e45ff374a3fb8105f7e6ee200bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } }
I20260619 01:06:29.832574 2661 tablet_replica.cc:442] T 086a7d1408d646a18b9fe392e783eeac P 18074e45ff374a3fb8105f7e6ee200bf: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:06:29.835032 2658 mvcc.cc:205] Tried to move back new op lower bound from 7298380553431216128 to 7298380553195302912. Current Snapshot: MvccSnapshot[applied={T|T < 7298380553431216128}]
I20260619 01:06:29.838037 2643 tablet_replica.cc:439] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18074e45ff374a3fb8105f7e6ee200bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } } }
I20260619 01:06:29.838387 2643 tablet_replica.cc:442] T 086a7d1408d646a18b9fe392e783eeac P 73602c8a32ab4f1ca995e6c5e1d1d9e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:06:29.848088 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.841943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.0.97.2:40229)
0619 01:06:29.842191 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.842197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.842536 (+ 339us) server_negotiation.cc:407] Connection header received
0619 01:06:29.842868 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.842872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.842959 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.843128 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:29.843525 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.844339 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.845818 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.846236 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.846378 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.846810 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:29.847427 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:29.847608 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.847690 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.847778 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 01:06:29.866609 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.860916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56881 (local address 127.0.97.1:38115)
0619 01:06:29.861160 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.861167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.861516 (+ 349us) server_negotiation.cc:407] Connection header received
0619 01:06:29.861760 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.861764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.861838 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.861941 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:29.862429 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.863243 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.864680 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.865145 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.865293 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.865811 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:29.865934 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:29.866046 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.866188 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.866270 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 01:06:29.866686 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.860733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38115 (local address 127.0.97.2:56881)
0619 01:06:29.861394 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.861417 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.861596 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.861957 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.861961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.861973 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:06:29.862281 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.862290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.863398 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.863404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.864507 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.864520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.865376 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.865384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.865521 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.865586 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:06:29.866046 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:06:29.866060 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.866165 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.866318 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":291,"threads_started":1}
I20260619 01:06:29.913071 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.907041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58743 (local address 127.0.97.2:40229)
0619 01:06:29.907241 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:29.907249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:29.907681 (+ 432us) server_negotiation.cc:407] Connection header received
0619 01:06:29.907851 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:29.907857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:29.907947 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:29.908052 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:29.908560 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.909650 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.911236 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.911675 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.911812 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.912280 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:29.912404 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:29.912506 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:29.912673 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 01:06:29.912756 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:06:29.913225 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:29.906886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40229 (local address 127.0.97.1:58743)
0619 01:06:29.907567 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:29.907584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:29.907715 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:29.908068 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:29.908071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:29.908081 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:06:29.908393 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.908401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.909789 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.909794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.911080 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:29.911093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:29.911866 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:29.911874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:29.912016 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:29.912078 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:06:29.912507 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:06:29.912521 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 01:06:29.912651 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 01:06:29.912799 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":321,"threads_started":1}
I20260619 01:06:29.958925 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:29.959259 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:29.963598 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a: Sending DeleteTablet for 3 replicas of tablet c340a84f1808446bbeae7f42df42e19d
I20260619 01:06:29.965220 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet c340a84f1808446bbeae7f42df42e19d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56230
I20260619 01:06:29.965517 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet c340a84f1808446bbeae7f42df42e19d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56736
I20260619 01:06:29.967561 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 01:06:29.967864 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet c340a84f1808446bbeae7f42df42e19d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40150
W20260619 01:06:29.968590 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:06:29.978039 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 3804305a19bc47e48e60fc7f0112c71a (DEFAULT_TABLE table=test-table [id=63e7036b5167438e9d9227d5e7686d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:29.978760 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3804305a19bc47e48e60fc7f0112c71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:29.979935 2686 tablet_replica.cc:330] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: stopping tablet replica
I20260619 01:06:29.981257 2686 raft_consensus.cc:2229] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:29.982025 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 3804305a19bc47e48e60fc7f0112c71a (DEFAULT_TABLE table=test-table [id=63e7036b5167438e9d9227d5e7686d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:29.982510 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3804305a19bc47e48e60fc7f0112c71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:29.982715 2686 raft_consensus.cc:2258] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:29.986567 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 3804305a19bc47e48e60fc7f0112c71a (DEFAULT_TABLE table=test-table [id=63e7036b5167438e9d9227d5e7686d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:29.987311 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3804305a19bc47e48e60fc7f0112c71a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:29.991532 2686 ts_tablet_manager.cc:1905] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:29.994988 2684 tablet_replica.cc:330] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: stopping tablet replica
I20260619 01:06:29.995445 2684 raft_consensus.cc:2229] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:06:29.996093 2684 raft_consensus.cc:2258] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:30.000401 2686 ts_tablet_manager.cc:1918] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:06:30.000597 2686 log.cc:1196] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/c340a84f1808446bbeae7f42df42e19d
I20260619 01:06:30.001199 2686 ts_tablet_manager.cc:1939] T c340a84f1808446bbeae7f42df42e19d P 18074e45ff374a3fb8105f7e6ee200bf: Deleting consensus metadata
I20260619 01:06:30.003173 2114 catalog_manager.cc:4841] TS 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1:38115): tablet c340a84f1808446bbeae7f42df42e19d (table test-table [id=3331479ca7114183bafc60eb037c2394]) successfully deleted
I20260619 01:06:30.005272 2684 ts_tablet_manager.cc:1905] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:30.014398 2684 ts_tablet_manager.cc:1918] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:06:30.014741 2684 log.cc:1196] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/c340a84f1808446bbeae7f42df42e19d
I20260619 01:06:30.015393 2684 ts_tablet_manager.cc:1939] T c340a84f1808446bbeae7f42df42e19d P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting consensus metadata
I20260619 01:06:30.018112 2114 catalog_manager.cc:4841] TS 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2:40229): tablet c340a84f1808446bbeae7f42df42e19d (table test-table [id=3331479ca7114183bafc60eb037c2394]) successfully deleted
I20260619 01:06:30.025826 2689 tablet_bootstrap.cc:492] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Bootstrap starting.
I20260619 01:06:30.025817 2688 tablet_bootstrap.cc:492] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb: Bootstrap starting.
I20260619 01:06:30.027954 2689 tablet_bootstrap.cc:654] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:30.027935 2688 tablet_bootstrap.cc:654] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:30.031742 2685 tablet_replica.cc:330] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: stopping tablet replica
I20260619 01:06:30.032259 2685 raft_consensus.cc:2229] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:30.032665 2685 raft_consensus.cc:2258] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:30.034745 2689 tablet_bootstrap.cc:492] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9: No bootstrap required, opened a new log
I20260619 01:06:30.034754 2685 ts_tablet_manager.cc:1905] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:30.035537 2688 tablet_bootstrap.cc:492] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb: No bootstrap required, opened a new log
I20260619 01:06:30.035797 2689 ts_tablet_manager.cc:1397] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 01:06:30.036559 2689 raft_consensus.cc:348] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:30.036787 2689 raft_consensus.cc:374] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:30.036966 2688 ts_tablet_manager.cc:1397] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260619 01:06:30.037429 2689 raft_consensus.cc:729] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73602c8a32ab4f1ca995e6c5e1d1d9e9, State: Initialized, Role: FOLLOWER
I20260619 01:06:30.037793 2689 consensus_queue.cc:260] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:30.037882 2688 raft_consensus.cc:348] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:30.038148 2688 raft_consensus.cc:374] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:30.038255 2688 raft_consensus.cc:729] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318291c09cef44cab20a45bf482185cb, State: Initialized, Role: FOLLOWER
I20260619 01:06:30.038313 2687 tablet_bootstrap.cc:492] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf: Bootstrap starting.
I20260619 01:06:30.038547 2688 consensus_queue.cc:260] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:30.039168 2689 ts_tablet_manager.cc:1428] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 01:06:30.039253 2688 ts_tablet_manager.cc:1428] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:06:30.044389 2687 tablet_bootstrap.cc:654] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:30.049701 2685 ts_tablet_manager.cc:1918] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:06:30.049913 2685 log.cc:1196] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/c340a84f1808446bbeae7f42df42e19d
I20260619 01:06:30.050472 2685 ts_tablet_manager.cc:1939] T c340a84f1808446bbeae7f42df42e19d P 318291c09cef44cab20a45bf482185cb: Deleting consensus metadata
I20260619 01:06:30.054008 2114 catalog_manager.cc:4841] TS 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637): tablet c340a84f1808446bbeae7f42df42e19d (table test-table [id=3331479ca7114183bafc60eb037c2394]) successfully deleted
I20260619 01:06:30.056005 2687 tablet_bootstrap.cc:492] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf: No bootstrap required, opened a new log
I20260619 01:06:30.056177 2687 ts_tablet_manager.cc:1397] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.006s
I20260619 01:06:30.057082 2687 raft_consensus.cc:348] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:30.057420 2687 raft_consensus.cc:374] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:30.057541 2687 raft_consensus.cc:729] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18074e45ff374a3fb8105f7e6ee200bf, State: Initialized, Role: FOLLOWER
I20260619 01:06:30.057807 2687 consensus_queue.cc:260] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:30.058583 2687 ts_tablet_manager.cc:1428] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 01:06:30.241074 2678 raft_consensus.cc:482] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:30.241380 2678 raft_consensus.cc:504] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:30.242722 2678 leader_election.cc:288] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1:38115), 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637)
I20260619 01:06:30.243613 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3804305a19bc47e48e60fc7f0112c71a" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18074e45ff374a3fb8105f7e6ee200bf" is_pre_election: true
I20260619 01:06:30.243651 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3804305a19bc47e48e60fc7f0112c71a" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318291c09cef44cab20a45bf482185cb" is_pre_election: true
I20260619 01:06:30.243898 2285 raft_consensus.cc:2454] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 0.
I20260619 01:06:30.243896 2563 raft_consensus.cc:2454] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 0.
I20260619 01:06:30.244562 2357 leader_election.cc:302] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18074e45ff374a3fb8105f7e6ee200bf, 73602c8a32ab4f1ca995e6c5e1d1d9e9; no voters:
I20260619 01:06:30.244958 2678 raft_consensus.cc:2781] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:06:30.245088 2678 raft_consensus.cc:482] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:06:30.245193 2678 raft_consensus.cc:3037] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:30.249214 2678 raft_consensus.cc:504] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:30.250121 2678 leader_election.cc:288] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 election: Requested vote from peers 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1:38115), 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637)
I20260619 01:06:30.250895 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3804305a19bc47e48e60fc7f0112c71a" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18074e45ff374a3fb8105f7e6ee200bf"
I20260619 01:06:30.250888 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3804305a19bc47e48e60fc7f0112c71a" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318291c09cef44cab20a45bf482185cb"
I20260619 01:06:30.251160 2563 raft_consensus.cc:3037] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:30.251165 2285 raft_consensus.cc:3037] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:30.254354 2563 raft_consensus.cc:2454] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 1.
I20260619 01:06:30.255232 2285 raft_consensus.cc:2454] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 1.
I20260619 01:06:30.255381 2357 leader_election.cc:302] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 318291c09cef44cab20a45bf482185cb, 73602c8a32ab4f1ca995e6c5e1d1d9e9; no voters:
I20260619 01:06:30.256337 2678 raft_consensus.cc:2781] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:30.256582 2678 raft_consensus.cc:686] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 LEADER]: Becoming Leader. State: Replica: 73602c8a32ab4f1ca995e6c5e1d1d9e9, State: Running, Role: LEADER
I20260619 01:06:30.256933 2678 consensus_queue.cc:237] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:30.260206 2128 catalog_manager.cc:5495] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } health_report { overall_health: UNKNOWN } } }
I20260619 01:06:30.341924 2285 raft_consensus.cc:1261] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Refusing update from remote peer 73602c8a32ab4f1ca995e6c5e1d1d9e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:30.341972 2563 raft_consensus.cc:1261] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Refusing update from remote peer 73602c8a32ab4f1ca995e6c5e1d1d9e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:30.342648 2678 consensus_queue.cc:1035] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:30.342934 2679 consensus_queue.cc:1035] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 01:06:30.382099 2658 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 01:06:31.979249 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:31.979578 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:31.983752 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a: Sending DeleteTablet for 3 replicas of tablet 3804305a19bc47e48e60fc7f0112c71a
I20260619 01:06:31.984737 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 3804305a19bc47e48e60fc7f0112c71a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40150
I20260619 01:06:31.984737 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 3804305a19bc47e48e60fc7f0112c71a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56230
I20260619 01:06:31.984822 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 3804305a19bc47e48e60fc7f0112c71a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56736
I20260619 01:06:31.997174 2717 tablet_replica.cc:330] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9: stopping tablet replica
I20260619 01:06:31.997557 2717 raft_consensus.cc:2229] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:06:31.998006 2717 raft_consensus.cc:2258] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:31.999126 2717 ts_tablet_manager.cc:1905] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:32.006821 2717 ts_tablet_manager.cc:1918] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 01:06:32.007097 2717 log.cc:1196] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/3804305a19bc47e48e60fc7f0112c71a
I20260619 01:06:32.007601 2717 ts_tablet_manager.cc:1939] T 3804305a19bc47e48e60fc7f0112c71a P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting consensus metadata
I20260619 01:06:32.008235 2718 tablet_replica.cc:330] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf: stopping tablet replica
I20260619 01:06:32.008322 2719 tablet_replica.cc:330] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb: stopping tablet replica
I20260619 01:06:32.008432 2718 raft_consensus.cc:2229] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:32.008522 2719 raft_consensus.cc:2229] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:32.008666 2718 raft_consensus.cc:2258] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:32.008766 2719 raft_consensus.cc:2258] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:32.009681 2114 catalog_manager.cc:4841] TS 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2:40229): tablet 3804305a19bc47e48e60fc7f0112c71a (table test-table [id=63e7036b5167438e9d9227d5e7686d84]) successfully deleted
I20260619 01:06:32.009781 2719 ts_tablet_manager.cc:1905] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:32.010154 2718 ts_tablet_manager.cc:1905] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:32.017567 2718 ts_tablet_manager.cc:1918] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 01:06:32.017763 2718 log.cc:1196] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/3804305a19bc47e48e60fc7f0112c71a
I20260619 01:06:32.017760 2719 ts_tablet_manager.cc:1918] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 01:06:32.017943 2719 log.cc:1196] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/3804305a19bc47e48e60fc7f0112c71a
I20260619 01:06:32.018201 2718 ts_tablet_manager.cc:1939] T 3804305a19bc47e48e60fc7f0112c71a P 18074e45ff374a3fb8105f7e6ee200bf: Deleting consensus metadata
I20260619 01:06:32.018422 2719 ts_tablet_manager.cc:1939] T 3804305a19bc47e48e60fc7f0112c71a P 318291c09cef44cab20a45bf482185cb: Deleting consensus metadata
I20260619 01:06:32.019541 2114 catalog_manager.cc:4841] TS 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1:38115): tablet 3804305a19bc47e48e60fc7f0112c71a (table test-table [id=63e7036b5167438e9d9227d5e7686d84]) successfully deleted
I20260619 01:06:32.019876 2114 catalog_manager.cc:4841] TS 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637): tablet 3804305a19bc47e48e60fc7f0112c71a (table test-table [id=63e7036b5167438e9d9227d5e7686d84]) successfully deleted
Jun 19 01:06:32 dist-test-slave-5p8w krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 01:06:32 dist-test-slave-5p8w krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:32.064251 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:32.038470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.0.97.62:39077)
0619 01:06:32.039645 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:32.039650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:32.039669 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:06:32.039720 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:32.039724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:32.039782 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:32.039937 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:32.041353 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:32.042205 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:32.043569 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:32.044053 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:32.044196 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:32.058396 (+ 14200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:32.058406 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:32.058412 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:32.058511 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:32.061391 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:32.062213 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:32.062221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:32.062229 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:32.062375 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:32.062775 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:32.062780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:32.062783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:32.063059 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:32.063330 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:32.063617 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 01:06:32.063859 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":914,"threads_started":1}
I20260619 01:06:32.067919 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:06:32.068727 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:06:32.077203 2264 tablet_service.cc:1467] Processing CreateTablet for tablet c830202693664461b94bb8255cba1c01 (DEFAULT_TABLE table=test-table [id=d47bfa540f6b4faebbe62f771c754709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:32.077373 2543 tablet_service.cc:1467] Processing CreateTablet for tablet c830202693664461b94bb8255cba1c01 (DEFAULT_TABLE table=test-table [id=d47bfa540f6b4faebbe62f771c754709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:32.077535 2403 tablet_service.cc:1467] Processing CreateTablet for tablet c830202693664461b94bb8255cba1c01 (DEFAULT_TABLE table=test-table [id=d47bfa540f6b4faebbe62f771c754709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:32.077762 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c830202693664461b94bb8255cba1c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:32.077785 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c830202693664461b94bb8255cba1c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:32.077967 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c830202693664461b94bb8255cba1c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:32.089756 2728 tablet_bootstrap.cc:492] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb: Bootstrap starting.
I20260619 01:06:32.090955 2730 tablet_bootstrap.cc:492] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf: Bootstrap starting.
I20260619 01:06:32.091336 2729 tablet_bootstrap.cc:492] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Bootstrap starting.
I20260619 01:06:32.091524 2728 tablet_bootstrap.cc:654] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:32.093171 2730 tablet_bootstrap.cc:654] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:32.093173 2729 tablet_bootstrap.cc:654] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:32.095640 2729 tablet_bootstrap.cc:492] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: No bootstrap required, opened a new log
I20260619 01:06:32.095876 2729 ts_tablet_manager.cc:1397] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 01:06:32.095934 2730 tablet_bootstrap.cc:492] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf: No bootstrap required, opened a new log
I20260619 01:06:32.096366 2728 tablet_bootstrap.cc:492] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb: No bootstrap required, opened a new log
I20260619 01:06:32.096534 2728 ts_tablet_manager.cc:1397] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260619 01:06:32.097337 2730 ts_tablet_manager.cc:1397] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 01:06:32.097337 2729 raft_consensus.cc:348] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.097735 2729 raft_consensus.cc:374] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:32.097882 2729 raft_consensus.cc:729] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73602c8a32ab4f1ca995e6c5e1d1d9e9, State: Initialized, Role: FOLLOWER
I20260619 01:06:32.097889 2728 raft_consensus.cc:348] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.098165 2728 raft_consensus.cc:374] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:32.098080 2730 raft_consensus.cc:348] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.098276 2728 raft_consensus.cc:729] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318291c09cef44cab20a45bf482185cb, State: Initialized, Role: FOLLOWER
I20260619 01:06:32.098317 2730 raft_consensus.cc:374] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:32.098399 2730 raft_consensus.cc:729] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18074e45ff374a3fb8105f7e6ee200bf, State: Initialized, Role: FOLLOWER
I20260619 01:06:32.098541 2728 consensus_queue.cc:260] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.098482 2729 consensus_queue.cc:260] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.098733 2730 consensus_queue.cc:260] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.099480 2730 ts_tablet_manager.cc:1428] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:06:32.101339 2729 ts_tablet_manager.cc:1428] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260619 01:06:32.105731 2728 ts_tablet_manager.cc:1428] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb: Time spent starting tablet: real 0.009s user 0.003s sys 0.006s
I20260619 01:06:32.353152 2708 raft_consensus.cc:482] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:32.353441 2708 raft_consensus.cc:504] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.354137 2708 leader_election.cc:288] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1:38115), 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637)
I20260619 01:06:32.354799 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c830202693664461b94bb8255cba1c01" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18074e45ff374a3fb8105f7e6ee200bf" is_pre_election: true
I20260619 01:06:32.354861 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c830202693664461b94bb8255cba1c01" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318291c09cef44cab20a45bf482185cb" is_pre_election: true
I20260619 01:06:32.355079 2285 raft_consensus.cc:2454] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 0.
I20260619 01:06:32.355216 2563 raft_consensus.cc:2454] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 0.
I20260619 01:06:32.355691 2357 leader_election.cc:302] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18074e45ff374a3fb8105f7e6ee200bf, 73602c8a32ab4f1ca995e6c5e1d1d9e9; no voters:
I20260619 01:06:32.356043 2708 raft_consensus.cc:2781] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:06:32.356281 2708 raft_consensus.cc:482] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:06:32.356407 2708 raft_consensus.cc:3037] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:32.358734 2708 raft_consensus.cc:504] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.359457 2708 leader_election.cc:288] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 election: Requested vote from peers 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1:38115), 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637)
I20260619 01:06:32.360146 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c830202693664461b94bb8255cba1c01" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18074e45ff374a3fb8105f7e6ee200bf"
I20260619 01:06:32.360260 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c830202693664461b94bb8255cba1c01" candidate_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318291c09cef44cab20a45bf482185cb"
I20260619 01:06:32.360395 2285 raft_consensus.cc:3037] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:32.360594 2563 raft_consensus.cc:3037] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:32.362676 2285 raft_consensus.cc:2454] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 1.
I20260619 01:06:32.362721 2563 raft_consensus.cc:2454] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73602c8a32ab4f1ca995e6c5e1d1d9e9 in term 1.
I20260619 01:06:32.363256 2357 leader_election.cc:302] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18074e45ff374a3fb8105f7e6ee200bf, 73602c8a32ab4f1ca995e6c5e1d1d9e9; no voters:
I20260619 01:06:32.363667 2708 raft_consensus.cc:2781] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:32.363845 2708 raft_consensus.cc:686] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 LEADER]: Becoming Leader. State: Replica: 73602c8a32ab4f1ca995e6c5e1d1d9e9, State: Running, Role: LEADER
I20260619 01:06:32.364182 2708 consensus_queue.cc:237] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.367105 2127 catalog_manager.cc:5495] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } health_report { overall_health: UNKNOWN } } }
I20260619 01:06:32.401758 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:32.396278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.0.97.2:40229)
0619 01:06:32.396985 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:32.396989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:32.397010 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:06:32.397069 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:32.397073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:32.397135 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:32.397219 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:32.397648 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:32.398537 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:32.399829 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:32.400343 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:32.400461 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:32.400926 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:32.401050 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:32.401122 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:32.401262 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 01:06:32.401399 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":446,"threads_started":1}
I20260619 01:06:32.412532 2285 raft_consensus.cc:1261] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Refusing update from remote peer 73602c8a32ab4f1ca995e6c5e1d1d9e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:32.412593 2563 raft_consensus.cc:1261] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Refusing update from remote peer 73602c8a32ab4f1ca995e6c5e1d1d9e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:32.413185 2679 consensus_queue.cc:1035] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:32.413552 2708 consensus_queue.cc:1035] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:32.422490 2699 mvcc.cc:205] Tried to move back new op lower bound from 7298380564115873792 to 7298380563926163456. Current Snapshot: MvccSnapshot[applied={T|T < 7298380564115873792}]
I20260619 01:06:32.463613 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:32.463858 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:32.466821 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a: Sending DeleteTablet for 3 replicas of tablet c830202693664461b94bb8255cba1c01
I20260619 01:06:32.467790 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet c830202693664461b94bb8255cba1c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40150
I20260619 01:06:32.467873 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet c830202693664461b94bb8255cba1c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56230
I20260619 01:06:32.467945 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet c830202693664461b94bb8255cba1c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56736
I20260619 01:06:32.468080 2718 tablet_replica.cc:330] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf: stopping tablet replica
I20260619 01:06:32.468194 2717 tablet_replica.cc:330] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: stopping tablet replica
I20260619 01:06:32.468303 2718 raft_consensus.cc:2229] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:32.468407 2719 tablet_replica.cc:330] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb: stopping tablet replica
I20260619 01:06:32.468394 2717 raft_consensus.cc:2229] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:06:32.468515 2718 raft_consensus.cc:2258] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:32.468604 2719 raft_consensus.cc:2229] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:32.468736 2717 raft_consensus.cc:2258] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:32.468778 2719 raft_consensus.cc:2258] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:32.468990 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 01:06:32.469743 2718 ts_tablet_manager.cc:1905] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 01:06:32.469799 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:06:32.470145 2717 ts_tablet_manager.cc:1905] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:32.470124 2719 ts_tablet_manager.cc:1905] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:32.476696 2718 ts_tablet_manager.cc:1918] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:06:32.476871 2718 log.cc:1196] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/c830202693664461b94bb8255cba1c01
I20260619 01:06:32.477267 2718 ts_tablet_manager.cc:1939] T c830202693664461b94bb8255cba1c01 P 18074e45ff374a3fb8105f7e6ee200bf: Deleting consensus metadata
I20260619 01:06:32.477617 2717 ts_tablet_manager.cc:1918] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:06:32.477787 2717 log.cc:1196] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/c830202693664461b94bb8255cba1c01
I20260619 01:06:32.478793 2403 tablet_service.cc:1467] Processing CreateTablet for tablet b9cb033822574e37acad001b888f6485 (DEFAULT_TABLE table=test-table [id=292fc897d77f4eef94781faa11da24a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:32.479000 2543 tablet_service.cc:1467] Processing CreateTablet for tablet b9cb033822574e37acad001b888f6485 (DEFAULT_TABLE table=test-table [id=292fc897d77f4eef94781faa11da24a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:32.479229 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9cb033822574e37acad001b888f6485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:32.479435 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9cb033822574e37acad001b888f6485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:32.481796 2719 ts_tablet_manager.cc:1918] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:06:32.481968 2719 log.cc:1196] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/c830202693664461b94bb8255cba1c01
I20260619 01:06:32.482429 2719 ts_tablet_manager.cc:1939] T c830202693664461b94bb8255cba1c01 P 318291c09cef44cab20a45bf482185cb: Deleting consensus metadata
I20260619 01:06:32.483453 2114 catalog_manager.cc:4841] TS 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637): tablet c830202693664461b94bb8255cba1c01 (table test-table [id=d47bfa540f6b4faebbe62f771c754709]) successfully deleted
I20260619 01:06:32.483697 2114 catalog_manager.cc:4841] TS 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1:38115): tablet c830202693664461b94bb8255cba1c01 (table test-table [id=d47bfa540f6b4faebbe62f771c754709]) successfully deleted
I20260619 01:06:32.484331 2264 tablet_service.cc:1467] Processing CreateTablet for tablet b9cb033822574e37acad001b888f6485 (DEFAULT_TABLE table=test-table [id=292fc897d77f4eef94781faa11da24a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:32.484725 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9cb033822574e37acad001b888f6485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:32.486013 2728 tablet_bootstrap.cc:492] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb: Bootstrap starting.
I20260619 01:06:32.487236 2717 ts_tablet_manager.cc:1939] T c830202693664461b94bb8255cba1c01 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting consensus metadata
I20260619 01:06:32.487303 2729 tablet_bootstrap.cc:492] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Bootstrap starting.
I20260619 01:06:32.488111 2728 tablet_bootstrap.cc:654] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:32.488766 2114 catalog_manager.cc:4841] TS 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2:40229): tablet c830202693664461b94bb8255cba1c01 (table test-table [id=d47bfa540f6b4faebbe62f771c754709]) successfully deleted
I20260619 01:06:32.488966 2729 tablet_bootstrap.cc:654] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:32.490819 2728 tablet_bootstrap.cc:492] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb: No bootstrap required, opened a new log
I20260619 01:06:32.491029 2728 ts_tablet_manager.cc:1397] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 01:06:32.491312 2730 tablet_bootstrap.cc:492] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf: Bootstrap starting.
I20260619 01:06:32.491850 2728 raft_consensus.cc:348] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.492136 2728 raft_consensus.cc:374] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:32.492226 2728 raft_consensus.cc:729] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318291c09cef44cab20a45bf482185cb, State: Initialized, Role: FOLLOWER
I20260619 01:06:32.492481 2729 tablet_bootstrap.cc:492] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: No bootstrap required, opened a new log
I20260619 01:06:32.492480 2728 consensus_queue.cc:260] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.492645 2729 ts_tablet_manager.cc:1397] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260619 01:06:32.493054 2728 ts_tablet_manager.cc:1428] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 01:06:32.493491 2730 tablet_bootstrap.cc:654] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:32.493459 2729 raft_consensus.cc:348] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.493707 2729 raft_consensus.cc:374] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:32.493799 2729 raft_consensus.cc:729] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73602c8a32ab4f1ca995e6c5e1d1d9e9, State: Initialized, Role: FOLLOWER
I20260619 01:06:32.494065 2729 consensus_queue.cc:260] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.494793 2729 ts_tablet_manager.cc:1428] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 01:06:32.496313 2730 tablet_bootstrap.cc:492] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf: No bootstrap required, opened a new log
I20260619 01:06:32.496522 2730 ts_tablet_manager.cc:1397] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 01:06:32.497253 2730 raft_consensus.cc:348] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.497603 2730 raft_consensus.cc:374] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:32.497690 2730 raft_consensus.cc:729] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18074e45ff374a3fb8105f7e6ee200bf, State: Initialized, Role: FOLLOWER
I20260619 01:06:32.497938 2730 consensus_queue.cc:260] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.498414 2730 ts_tablet_manager.cc:1428] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 01:06:32.505501 2703 raft_consensus.cc:482] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:32.505751 2703 raft_consensus.cc:504] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.506538 2703 leader_election.cc:288] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2:40229), 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637)
I20260619 01:06:32.507107 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9cb033822574e37acad001b888f6485" candidate_uuid: "18074e45ff374a3fb8105f7e6ee200bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" is_pre_election: true
I20260619 01:06:32.507206 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9cb033822574e37acad001b888f6485" candidate_uuid: "18074e45ff374a3fb8105f7e6ee200bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318291c09cef44cab20a45bf482185cb" is_pre_election: true
I20260619 01:06:32.507439 2563 raft_consensus.cc:2454] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18074e45ff374a3fb8105f7e6ee200bf in term 0.
I20260619 01:06:32.507439 2423 raft_consensus.cc:2454] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18074e45ff374a3fb8105f7e6ee200bf in term 0.
I20260619 01:06:32.507923 2217 leader_election.cc:302] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18074e45ff374a3fb8105f7e6ee200bf, 318291c09cef44cab20a45bf482185cb; no voters:
I20260619 01:06:32.508249 2703 raft_consensus.cc:2781] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:06:32.508354 2703 raft_consensus.cc:482] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:06:32.508464 2703 raft_consensus.cc:3037] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:32.510950 2703 raft_consensus.cc:504] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.511524 2703 leader_election.cc:288] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [CANDIDATE]: Term 1 election: Requested vote from peers 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2:40229), 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637)
I20260619 01:06:32.512146 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9cb033822574e37acad001b888f6485" candidate_uuid: "18074e45ff374a3fb8105f7e6ee200bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9"
I20260619 01:06:32.512215 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9cb033822574e37acad001b888f6485" candidate_uuid: "18074e45ff374a3fb8105f7e6ee200bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318291c09cef44cab20a45bf482185cb"
I20260619 01:06:32.512426 2563 raft_consensus.cc:3037] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:32.512419 2423 raft_consensus.cc:3037] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:32.515529 2423 raft_consensus.cc:2454] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18074e45ff374a3fb8105f7e6ee200bf in term 1.
I20260619 01:06:32.515529 2563 raft_consensus.cc:2454] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18074e45ff374a3fb8105f7e6ee200bf in term 1.
I20260619 01:06:32.516103 2217 leader_election.cc:302] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18074e45ff374a3fb8105f7e6ee200bf, 73602c8a32ab4f1ca995e6c5e1d1d9e9; no voters:
I20260619 01:06:32.516461 2703 raft_consensus.cc:2781] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:32.516669 2703 raft_consensus.cc:686] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 1 LEADER]: Becoming Leader. State: Replica: 18074e45ff374a3fb8105f7e6ee200bf, State: Running, Role: LEADER
I20260619 01:06:32.517017 2703 consensus_queue.cc:237] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } }
I20260619 01:06:32.520179 2127 catalog_manager.cc:5495] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1). New cstate: current_term: 1 leader_uuid: "18074e45ff374a3fb8105f7e6ee200bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18074e45ff374a3fb8105f7e6ee200bf" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 } health_report { overall_health: UNKNOWN } } }
I20260619 01:06:32.551403 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:32.544923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.0.97.1:38115)
0619 01:06:32.545733 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:32.545738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:32.545758 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:06:32.545817 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:32.545821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:32.545884 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:32.545970 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:32.546471 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:32.547245 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:32.549275 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:32.549777 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:32.549889 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:32.550483 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:32.550620 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:32.550708 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:32.550861 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 01:06:32.550944 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":548,"threads_started":1}
I20260619 01:06:32.568522 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:32.562556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36445 (local address 127.0.97.1:38115)
0619 01:06:32.562737 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:32.562742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:32.563293 (+ 551us) server_negotiation.cc:407] Connection header received
0619 01:06:32.563525 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:32.563530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:32.563600 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:32.563707 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:32.564146 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:32.564990 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:32.566578 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:32.567015 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:32.567168 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:32.567597 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:32.567712 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:32.567820 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:32.567980 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 01:06:32.568052 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 01:06:32.568595 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:32.562481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38115 (local address 127.0.97.1:36445)
0619 01:06:32.563168 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:32.563185 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:32.563386 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:32.563718 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:32.563722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:32.563733 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:06:32.563997 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:32.564007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:32.565141 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:32.565148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:32.566411 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:32.566426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:32.567221 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:32.567230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:32.567387 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:32.567428 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:06:32.567818 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:06:32.567841 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 01:06:32.567945 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 01:06:32.568094 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":423,"threads_started":1}
I20260619 01:06:32.577847 2563 raft_consensus.cc:1261] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Refusing update from remote peer 18074e45ff374a3fb8105f7e6ee200bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:32.577849 2423 raft_consensus.cc:1261] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Refusing update from remote peer 18074e45ff374a3fb8105f7e6ee200bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:32.578469 2663 consensus_queue.cc:1035] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "73602c8a32ab4f1ca995e6c5e1d1d9e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:32.578864 2703 consensus_queue.cc:1035] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "318291c09cef44cab20a45bf482185cb" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 01:06:32.601282 2696 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 01:06:34.203303 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:34.203549 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:34.208889 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c31350d3c3714f82b2d3266cdce9049a: Sending DeleteTablet for 3 replicas of tablet b9cb033822574e37acad001b888f6485
I20260619 01:06:34.210074 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet b9cb033822574e37acad001b888f6485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40150
I20260619 01:06:34.210207 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet b9cb033822574e37acad001b888f6485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56230
I20260619 01:06:34.210244 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet b9cb033822574e37acad001b888f6485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56736
I20260619 01:06:34.210958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 2199
I20260619 01:06:34.222862 2773 tablet_replica.cc:330] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb: stopping tablet replica
I20260619 01:06:34.223248 2773 raft_consensus.cc:2229] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:34.223611 2773 raft_consensus.cc:2258] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:34.224397 2772 tablet_replica.cc:330] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: stopping tablet replica
I20260619 01:06:34.224603 2771 tablet_replica.cc:330] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf: stopping tablet replica
I20260619 01:06:34.224786 2772 raft_consensus.cc:2229] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:34.224910 2771 raft_consensus.cc:2229] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:06:34.225395 2772 raft_consensus.cc:2258] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:34.225395 2773 ts_tablet_manager.cc:1905] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:34.225670 2771 raft_consensus.cc:2258] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:34.226677 2772 ts_tablet_manager.cc:1905] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:34.227023 2771 ts_tablet_manager.cc:1905] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:34.233140 2773 ts_tablet_manager.cc:1918] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 01:06:34.233357 2773 log.cc:1196] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/b9cb033822574e37acad001b888f6485
I20260619 01:06:34.233861 2773 ts_tablet_manager.cc:1939] T b9cb033822574e37acad001b888f6485 P 318291c09cef44cab20a45bf482185cb: Deleting consensus metadata
I20260619 01:06:34.234477 2772 ts_tablet_manager.cc:1918] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 01:06:34.234702 2772 log.cc:1196] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/b9cb033822574e37acad001b888f6485
I20260619 01:06:34.235109 2771 ts_tablet_manager.cc:1918] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 01:06:34.235248 2771 log.cc:1196] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/b9cb033822574e37acad001b888f6485
I20260619 01:06:34.235249 2114 catalog_manager.cc:4841] TS 318291c09cef44cab20a45bf482185cb (127.0.97.3:43637): tablet b9cb033822574e37acad001b888f6485 (table test-table [id=292fc897d77f4eef94781faa11da24a3]) successfully deleted
I20260619 01:06:34.235361 2772 ts_tablet_manager.cc:1939] T b9cb033822574e37acad001b888f6485 P 73602c8a32ab4f1ca995e6c5e1d1d9e9: Deleting consensus metadata
I20260619 01:06:34.235728 2771 ts_tablet_manager.cc:1939] T b9cb033822574e37acad001b888f6485 P 18074e45ff374a3fb8105f7e6ee200bf: Deleting consensus metadata
I20260619 01:06:34.236905 2114 catalog_manager.cc:4841] TS 73602c8a32ab4f1ca995e6c5e1d1d9e9 (127.0.97.2:40229): tablet b9cb033822574e37acad001b888f6485 (table test-table [id=292fc897d77f4eef94781faa11da24a3]) successfully deleted
I20260619 01:06:34.237195 2114 catalog_manager.cc:4841] TS 18074e45ff374a3fb8105f7e6ee200bf (127.0.97.1:38115): tablet b9cb033822574e37acad001b888f6485 (table test-table [id=292fc897d77f4eef94781faa11da24a3]) successfully deleted
Jun 19 01:06:34 dist-test-slave-5p8w krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:06:34.248447 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:34.212359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.0.97.1:38115)
0619 01:06:34.230612 (+ 18253us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:34.230618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:34.230652 (+ 34us) server_negotiation.cc:407] Connection header received
0619 01:06:34.230715 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:34.230720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:34.230804 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:34.232955 (+ 2151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:34.233165 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:34.234404 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:34.236489 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:34.237147 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:34.237253 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:34.242441 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:34.242458 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:34.242523 (+ 65us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:34.242583 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:34.245811 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:34.246553 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:34.246560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:34.246563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:34.246695 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:34.247085 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:34.247090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:34.247092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:34.247488 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:34.247735 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:34.247956 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 01:06:34.248140 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18104,"thread_start_us":18618,"threads_started":1}
W20260619 01:06:34.249224 2330 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40172
W20260619 01:06:34.249986 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 2199: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:06:34.250169 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 2199
I20260619 01:06:34.277770 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 2339
Jun 19 01:06:34 dist-test-slave-5p8w krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:06:34.294296 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:34.278558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.0.97.2:40229)
0619 01:06:34.279080 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:34.279086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:34.279111 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:06:34.279167 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:34.279170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:34.279229 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:34.279351 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:34.280647 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:34.281566 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:34.282884 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:34.283297 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:34.283440 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:34.287898 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:34.287915 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:34.287931 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:34.287987 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:34.291262 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:34.292070 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:34.292087 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:34.292089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:34.292243 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:34.292651 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:34.292656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:34.292660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:34.293026 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:34.293373 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:34.293697 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 01:06:34.293921 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":262,"threads_started":1}
W20260619 01:06:34.295007 2469 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56266
W20260619 01:06:34.295667 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 2339: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:06:34.295871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 2339
I20260619 01:06:34.318481 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 2478
Jun 19 01:06:34 dist-test-slave-5p8w krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:06:34.335450 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:34.319318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.0.97.3:43637)
0619 01:06:34.319831 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:34.319836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:34.319853 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:06:34.319908 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:34.319911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:34.319991 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:34.320087 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:34.321709 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:34.322460 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:34.323800 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:34.324204 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:34.324301 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:34.328827 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:34.328843 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:34.328860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:34.328900 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:34.332385 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:34.333207 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:34.333214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:34.333217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:34.333460 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:34.333998 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:34.334003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:34.334006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:34.334500 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:34.334799 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:34.334996 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 01:06:34.335182 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":253,"threads_started":1}
W20260619 01:06:34.336191 2608 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56748
W20260619 01:06:34.336887 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 2478: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:06:34.337059 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 2478
I20260619 01:06:34.361454 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 2096
I20260619 01:06:34.362807 2168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:34.485946 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 2096
2026-06-19T01:06:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9920 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:06:34 dist-test-slave-5p8w krb5kdc[2781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:06:34 dist-test-slave-5p8w krb5kdc[2781](info): set up 2 sockets
Jun 19 01:06:34 dist-test-slave-5p8w krb5kdc[2781](info): commencing operation
krb5kdc: starting...
W20260619 01:06:35.585773 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:06:35 dist-test-slave-5p8w krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:06:35.786929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38005
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:36.135982 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:36.136300 2794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:36.136459 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:36.143947 2794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:36.144083 2794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:36.144186 2794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:36.144294 2794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:36.144356 2794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:06:36.154700 2794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:36.156250 2794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:36.157718 2794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:36.170218 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:36.170167 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:36.170125 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:36.171273 2794 server_base.cc:1034] running on GCE node
I20260619 01:06:36.172252 2794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:36.174005 2794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:36.175272 2794 hybrid_clock.cc:648] HybridClock initialized: now 1781831196175221 us; error 52 us; skew 500 ppm
I20260619 01:06:36.175673 2794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:36 dist-test-slave-5p8w krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:36.178999 2794 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:06:36.180691 2794 webserver.cc:460] Webserver started at http://127.0.97.62:39703/ using document root <none> and password file <none>
I20260619 01:06:36.181555 2794 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:36.181708 2794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:36.182044 2794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:36.185245 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "c13fe36190264547ad2e113a534c50c8"
format_stamp: "Formatted at 2026-06-19 01:06:36 on dist-test-slave-5p8w"
server_key: "48a6c500f67b261828be7c549d8e6184"
server_key_iv: "95de545833f1414dbdb2b32c0d34b239"
server_key_version: "encryptionkey@0"
I20260619 01:06:36.186221 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "c13fe36190264547ad2e113a534c50c8"
format_stamp: "Formatted at 2026-06-19 01:06:36 on dist-test-slave-5p8w"
server_key: "48a6c500f67b261828be7c549d8e6184"
server_key_iv: "95de545833f1414dbdb2b32c0d34b239"
server_key_version: "encryptionkey@0"
I20260619 01:06:36.191977 2794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:06:36.196147 2809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:36.197845 2794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:36.198087 2794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "c13fe36190264547ad2e113a534c50c8"
format_stamp: "Formatted at 2026-06-19 01:06:36 on dist-test-slave-5p8w"
server_key: "48a6c500f67b261828be7c549d8e6184"
server_key_iv: "95de545833f1414dbdb2b32c0d34b239"
server_key_version: "encryptionkey@0"
I20260619 01:06:36.198321 2794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:36.221743 2794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:36.226459 2794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:36.226804 2794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:36.246217 2794 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38005
I20260619 01:06:36.246245 2860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38005 every 8 connection(s)
I20260619 01:06:36.247820 2794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:36.251940 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:36.257992 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 2794
I20260619 01:06:36.258221 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:36.258870 388 external_mini_cluster.cc:1428] Setting key 628cef2adc510c320294567eb7a44bae
I20260619 01:06:36.264734 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8: Bootstrap starting.
I20260619 01:06:36.269124 2861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:36.270464 2861 log.cc:824] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:36.274066 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8: No bootstrap required, opened a new log
Jun 19 01:06:36 dist-test-slave-5p8w krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:36.285883 2861 raft_consensus.cc:348] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13fe36190264547ad2e113a534c50c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38005 } }
I20260619 01:06:36.286453 2861 raft_consensus.cc:374] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:36.286660 2861 raft_consensus.cc:729] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13fe36190264547ad2e113a534c50c8, State: Initialized, Role: FOLLOWER
I20260619 01:06:36.287513 2861 consensus_queue.cc:260] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13fe36190264547ad2e113a534c50c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38005 } }
I20260619 01:06:36.287839 2861 raft_consensus.cc:388] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:36.288010 2861 raft_consensus.cc:482] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:36.288230 2861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:36.288702 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:36.261188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.0.97.62:38005)
0619 01:06:36.262291 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:36.262309 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:36.262378 (+ 69us) server_negotiation.cc:407] Connection header received
0619 01:06:36.264914 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:36.264952 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:36.265626 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:36.266140 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:36.267694 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:36.269809 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:36.271593 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:36.272659 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:36.272866 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:36.277261 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:36.277273 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:36.277334 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:36.277399 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:36.282580 (+ 5181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:36.284001 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:36.284015 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:36.284027 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:36.284364 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:36.285040 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:36.285046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:36.285049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:36.285943 (+ 894us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:36.286282 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:36.286845 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 01:06:36.287348 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":469,"threads_started":1}
I20260619 01:06:36.291286 2861 raft_consensus.cc:504] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13fe36190264547ad2e113a534c50c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38005 } }
I20260619 01:06:36.291910 2861 leader_election.cc:302] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c13fe36190264547ad2e113a534c50c8; no voters:
I20260619 01:06:36.292706 2861 leader_election.cc:288] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:36.292809 2866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:36.293606 2866 raft_consensus.cc:686] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [term 1 LEADER]: Becoming Leader. State: Replica: c13fe36190264547ad2e113a534c50c8, State: Running, Role: LEADER
I20260619 01:06:36.294323 2866 consensus_queue.cc:237] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13fe36190264547ad2e113a534c50c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38005 } }
I20260619 01:06:36.295352 2861 sys_catalog.cc:564] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:36.297529 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c13fe36190264547ad2e113a534c50c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13fe36190264547ad2e113a534c50c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38005 } } }
I20260619 01:06:36.298058 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:36.297523 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c13fe36190264547ad2e113a534c50c8. Latest consensus state: current_term: 1 leader_uuid: "c13fe36190264547ad2e113a534c50c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c13fe36190264547ad2e113a534c50c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38005 } } }
I20260619 01:06:36.298539 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:36.302042 2872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:36.309859 2872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:36.323736 2872 catalog_manager.cc:1347] Generated new cluster ID: e2e0a9d75d1e40d3a7a5819a16a05422
I20260619 01:06:36.323894 2872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:36.349663 2872 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:36.351220 2872 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:36.364750 2872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c13fe36190264547ad2e113a534c50c8: Generated new TSK 0
I20260619 01:06:36.366070 2872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:06:36.421285 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38005
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:36.775796 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:36.776122 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:36.776217 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:36.783890 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:36.784031 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:36.784149 2887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:06:36.794745 2887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:36.796485 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:36.798031 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:36.812469 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:36.812608 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:36.812649 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:36.813409 2887 server_base.cc:1034] running on GCE node
I20260619 01:06:36.814201 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:36.815230 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:36.816447 2887 hybrid_clock.cc:648] HybridClock initialized: now 1781831196816398 us; error 46 us; skew 500 ppm
I20260619 01:06:36.816871 2887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:36 dist-test-slave-5p8w krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:36.820930 2887 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:06:36.822570 2887 webserver.cc:460] Webserver started at http://127.0.97.1:33313/ using document root <none> and password file <none>
I20260619 01:06:36.823359 2887 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:36.823515 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:36.823873 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:36.827093 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "1d7038fc84114d21b0944a0da5efbaf5"
format_stamp: "Formatted at 2026-06-19 01:06:36 on dist-test-slave-5p8w"
server_key: "1ff303d7e2fa6501d1ad987a7e7fcf51"
server_key_iv: "1cd58b1428a2e14c84b6879c9e72ff3d"
server_key_version: "encryptionkey@0"
I20260619 01:06:36.827986 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1d7038fc84114d21b0944a0da5efbaf5"
format_stamp: "Formatted at 2026-06-19 01:06:36 on dist-test-slave-5p8w"
server_key: "1ff303d7e2fa6501d1ad987a7e7fcf51"
server_key_iv: "1cd58b1428a2e14c84b6879c9e72ff3d"
server_key_version: "encryptionkey@0"
I20260619 01:06:36.833827 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 01:06:36.837602 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:36.839094 2887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:06:36.839308 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "1d7038fc84114d21b0944a0da5efbaf5"
format_stamp: "Formatted at 2026-06-19 01:06:36 on dist-test-slave-5p8w"
server_key: "1ff303d7e2fa6501d1ad987a7e7fcf51"
server_key_iv: "1cd58b1428a2e14c84b6879c9e72ff3d"
server_key_version: "encryptionkey@0"
I20260619 01:06:36.839567 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:36.874361 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:36.879292 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:36.879637 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:36.880856 2887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:36.882371 2887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:36.882493 2887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:36.882659 2887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:36.882737 2887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:36.917855 2887 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37805
I20260619 01:06:36.917919 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37805 every 8 connection(s)
I20260619 01:06:36.919436 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:06:36.923470 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 2887
I20260619 01:06:36.923753 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:06:36.926913 388 external_mini_cluster.cc:1428] Setting key 35d929fdc8d04f2bfb87b2505455e57b
Jun 19 01:06:36 dist-test-slave-5p8w krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:36.946378 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:36.922323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43269 (local address 127.0.97.62:38005)
0619 01:06:36.922911 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:36.922918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:36.927369 (+ 4451us) server_negotiation.cc:407] Connection header received
0619 01:06:36.927452 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:36.927457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:36.927548 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:36.927673 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:36.930091 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:36.931345 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:36.934348 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:36.934792 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:36.934922 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:36.939535 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:36.939541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:36.939544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:36.939581 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:36.942293 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:36.943544 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:36.943549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:36.943551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:36.943763 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:36.944395 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:36.944400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:36.944404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:36.944718 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:36.944900 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:36.945936 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0619 01:06:36.946123 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":319,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:06:36.948037 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:36.923005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38005 (local address 127.0.97.1:43269)
0619 01:06:36.923981 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:36.924041 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:36.926864 (+ 2823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:36.928023 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:36.928037 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:36.928855 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:36.929841 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:36.929873 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:36.931607 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:36.931616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:36.934123 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:36.934148 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:36.935090 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:36.935100 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:36.935425 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:36.936212 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:36.936269 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:36.939209 (+ 2940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:36.942537 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:36.942552 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:36.942580 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:36.943320 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:36.943900 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:36.943908 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:36.943915 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:36.944236 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:36.944878 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:36.944889 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:36.945452 (+ 563us) client_negotiation.cc:769] Sending connection context
0619 01:06:36.945821 (+ 369us) client_negotiation.cc:240] Negotiation successful
0619 01:06:36.946513 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":375,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:06:36.949909 3015 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38005
I20260619 01:06:36.950243 3015 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:36.951064 3015 heartbeater.cc:511] Master 127.0.97.62:38005 requested a full tablet report, sending...
I20260619 01:06:36.953469 2826 ts_manager.cc:194] Registered new tserver with Master: 1d7038fc84114d21b0944a0da5efbaf5 (127.0.97.1:37805)
I20260619 01:06:36.955572 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:06:36.976498 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38005
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:37.325343 3021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:37.325671 3021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:37.325841 3021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:37.333436 3021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:37.333566 3021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:37.333752 3021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:06:37.344242 3021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:37.345870 3021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:37.347271 3021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:37.362756 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:37.362737 3026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:37.362906 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:37.363579 3021 server_base.cc:1034] running on GCE node
I20260619 01:06:37.364224 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:37.365373 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:37.366690 3021 hybrid_clock.cc:648] HybridClock initialized: now 1781831197366602 us; error 71 us; skew 500 ppm
I20260619 01:06:37.367049 3021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:37 dist-test-slave-5p8w krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:37.375560 3021 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:06:37.377266 3021 webserver.cc:460] Webserver started at http://127.0.97.2:46649/ using document root <none> and password file <none>
I20260619 01:06:37.378085 3021 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:37.378193 3021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:37.378481 3021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:37.381668 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "609276e65b014bd9befb554d4564ac6e"
format_stamp: "Formatted at 2026-06-19 01:06:37 on dist-test-slave-5p8w"
server_key: "66ac0c5c404e3fe3b9622f1608e48a2a"
server_key_iv: "8de286494a0e8e8f63c28bb23554d61f"
server_key_version: "encryptionkey@0"
I20260619 01:06:37.382507 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "609276e65b014bd9befb554d4564ac6e"
format_stamp: "Formatted at 2026-06-19 01:06:37 on dist-test-slave-5p8w"
server_key: "66ac0c5c404e3fe3b9622f1608e48a2a"
server_key_iv: "8de286494a0e8e8f63c28bb23554d61f"
server_key_version: "encryptionkey@0"
I20260619 01:06:37.388348 3021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 01:06:37.392283 3036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:37.393987 3021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:37.394155 3021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "609276e65b014bd9befb554d4564ac6e"
format_stamp: "Formatted at 2026-06-19 01:06:37 on dist-test-slave-5p8w"
server_key: "66ac0c5c404e3fe3b9622f1608e48a2a"
server_key_iv: "8de286494a0e8e8f63c28bb23554d61f"
server_key_version: "encryptionkey@0"
I20260619 01:06:37.394375 3021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:37.412900 3021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:37.417891 3021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:37.418154 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:37.419152 3021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:37.420615 3021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:37.420696 3021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:37.420800 3021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:37.420850 3021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:37.457136 3021 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38489
I20260619 01:06:37.457172 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38489 every 8 connection(s)
I20260619 01:06:37.458571 3021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:06:37.467005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 3021
I20260619 01:06:37.467161 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:06:37.467718 388 external_mini_cluster.cc:1428] Setting key 4c8626766a6415c99348053c22cea000
Jun 19 01:06:37 dist-test-slave-5p8w krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:37.482138 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:37.461745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35429 (local address 127.0.97.62:38005)
0619 01:06:37.462215 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:37.462220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:37.463404 (+ 1184us) server_negotiation.cc:407] Connection header received
0619 01:06:37.465425 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:37.465432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:37.465530 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:37.465668 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:37.467535 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:37.468355 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:37.470430 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:37.470869 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:37.471023 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:37.475084 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:37.475094 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:37.475099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:37.475149 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:37.478301 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:37.479498 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:37.479502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:37.479505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:37.479649 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:37.480207 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:37.480210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:37.480212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:37.480482 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:37.480598 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:37.481649 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0619 01:06:37.481846 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":223,"threads_started":1}
I20260619 01:06:37.483922 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:37.462420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38005 (local address 127.0.97.2:35429)
0619 01:06:37.463205 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:37.463255 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:37.465064 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:37.465953 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:37.465963 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:37.466573 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:37.467315 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:37.467351 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:37.468541 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:37.468554 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:37.470269 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:37.470282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:37.471131 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:37.471139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:37.471392 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:37.472184 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:37.472257 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:37.474796 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:37.478498 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:37.478514 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:37.478538 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:37.479323 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:37.479775 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:37.479783 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:37.479790 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:37.480084 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:37.480609 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:37.480620 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:37.481158 (+ 538us) client_negotiation.cc:769] Sending connection context
0619 01:06:37.481546 (+ 388us) client_negotiation.cc:240] Negotiation successful
0619 01:06:37.482215 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":308,"threads_started":1}
I20260619 01:06:37.485615 3149 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38005
I20260619 01:06:37.485980 3149 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:37.486987 3149 heartbeater.cc:511] Master 127.0.97.62:38005 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:06:37.488917 2826 ts_manager.cc:194] Registered new tserver with Master: 609276e65b014bd9befb554d4564ac6e (127.0.97.2:38489)
I20260619 01:06:37.489887 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:06:37.517544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38005
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:37.868063 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:37.868448 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:37.868628 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:37.876124 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:37.876317 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:37.876504 3155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:06:37.886679 3155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:37.888069 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:37.889648 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:37.904698 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:37.904752 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:37.904618 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:37.905927 3155 server_base.cc:1034] running on GCE node
I20260619 01:06:37.906904 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:37.908041 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:37.909359 3155 hybrid_clock.cc:648] HybridClock initialized: now 1781831197909246 us; error 59 us; skew 500 ppm
I20260619 01:06:37.909741 3155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:37 dist-test-slave-5p8w krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:37.913061 3155 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:06:37.914654 3155 webserver.cc:460] Webserver started at http://127.0.97.3:40987/ using document root <none> and password file <none>
I20260619 01:06:37.915436 3155 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:37.915545 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:37.915908 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:37.919149 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "071e33b3ff1f428c8b412103fa8ac6f2"
format_stamp: "Formatted at 2026-06-19 01:06:37 on dist-test-slave-5p8w"
server_key: "fbdfbdd9c82720edd886dcfe67ce0797"
server_key_iv: "e0cc70253747ae12ecc41cee5af32431"
server_key_version: "encryptionkey@0"
I20260619 01:06:37.920042 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "071e33b3ff1f428c8b412103fa8ac6f2"
format_stamp: "Formatted at 2026-06-19 01:06:37 on dist-test-slave-5p8w"
server_key: "fbdfbdd9c82720edd886dcfe67ce0797"
server_key_iv: "e0cc70253747ae12ecc41cee5af32431"
server_key_version: "encryptionkey@0"
I20260619 01:06:37.926060 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 01:06:37.930192 3170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:37.931859 3155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:06:37.932055 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "071e33b3ff1f428c8b412103fa8ac6f2"
format_stamp: "Formatted at 2026-06-19 01:06:37 on dist-test-slave-5p8w"
server_key: "fbdfbdd9c82720edd886dcfe67ce0797"
server_key_iv: "e0cc70253747ae12ecc41cee5af32431"
server_key_version: "encryptionkey@0"
I20260619 01:06:37.932309 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:37.951246 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:37.955904 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:37.956238 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:37.957320 3155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:37.958740 3155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:37.958850 3155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:37.958982 3155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:37.959045 3155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:37.959108 3015 heartbeater.cc:503] Master 127.0.97.62:38005 was elected leader, sending a full tablet report...
I20260619 01:06:37.993876 3155 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45309
I20260619 01:06:37.993927 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45309 every 8 connection(s)
I20260619 01:06:37.995314 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:06:37.997603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 3155
I20260619 01:06:37.997807 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:06:37.998431 388 external_mini_cluster.cc:1428] Setting key d1f597f3e20d0ac7f2acf6d44de42dbd
Jun 19 01:06:38 dist-test-slave-5p8w krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:38.018301 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:37.998348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40633 (local address 127.0.97.62:38005)
0619 01:06:37.998871 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:37.998876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:38.000204 (+ 1328us) server_negotiation.cc:407] Connection header received
0619 01:06:38.002668 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:38.002673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:38.002782 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:38.002930 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:38.004568 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:38.005460 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:38.007129 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:38.007553 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:38.007693 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:38.011562 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:38.011567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:38.011571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:38.011608 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:38.015105 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:38.015975 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:38.015980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:38.015982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:38.016112 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:38.016546 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:38.016549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:38.016551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:38.016837 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:38.016957 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:38.017823 (+ 866us) server_negotiation.cc:299] Negotiation successful
0619 01:06:38.018015 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":309,"threads_started":1}
I20260619 01:06:38.019351 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:37.999043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38005 (local address 127.0.97.3:40633)
0619 01:06:38.000048 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:38.000092 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:38.002374 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:38.003197 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:38.003207 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:38.003756 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:38.004380 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:38.004398 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:38.005621 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:38.005627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:38.006968 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:38.006981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:38.007801 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:38.007812 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:38.008077 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:38.008852 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:38.008881 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:38.011291 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:38.015256 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:38.015265 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:38.015282 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:38.015808 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:38.016218 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:38.016222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:38.016224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:38.016446 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:38.016949 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:38.016956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:38.017425 (+ 469us) client_negotiation.cc:769] Sending connection context
0619 01:06:38.017685 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 01:06:38.018117 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":429,"threads_started":1}
I20260619 01:06:38.021185 3283 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38005
I20260619 01:06:38.021611 3283 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:38.022436 3283 heartbeater.cc:511] Master 127.0.97.62:38005 requested a full tablet report, sending...
I20260619 01:06:38.024168 2826 ts_manager.cc:194] Registered new tserver with Master: 071e33b3ff1f428c8b412103fa8ac6f2 (127.0.97.3:45309)
I20260619 01:06:38.024988 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40633
I20260619 01:06:38.027320 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 01:06:38.061282 3285 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:06:38.058502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.0.97.62:38005)
0619 01:06:38.058744 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:38.058753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:38.059167 (+ 414us) server_negotiation.cc:407] Connection header received
0619 01:06:38.059401 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:38.059405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:38.059489 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:38.059604 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:38.061003 (+ 1399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53650: BlockingRecv error: recv got EOF from 127.0.0.1:53650 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 01:06:38.064524 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 2887
W20260619 01:06:38.067970 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 2887: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:37805: server requires authentication, but client does not have Kerberos credentials available
I20260619 01:06:38.068198 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 2887
I20260619 01:06:38.084604 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3021
W20260619 01:06:38.087515 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3021: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:38489: server requires authentication, but client does not have Kerberos credentials available
I20260619 01:06:38.087689 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 3021
I20260619 01:06:38.103479 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3155
W20260619 01:06:38.107020 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3155: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45309: server requires authentication, but client does not have Kerberos credentials available
I20260619 01:06:38.107210 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 3155
I20260619 01:06:38.122876 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 2794
I20260619 01:06:38.124258 2856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:38.228613 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 2794
2026-06-19T01:06:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3740 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:06:38 dist-test-slave-5p8w krb5kdc[3301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:06:38 dist-test-slave-5p8w krb5kdc[3301](info): set up 2 sockets
Jun 19 01:06:38 dist-test-slave-5p8w krb5kdc[3301](info): commencing operation
krb5kdc: starting...
W20260619 01:06:39.325814 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:06:39 dist-test-slave-5p8w krb5kdc[3301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:06:39.532473 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34573
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:39.892333 3314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:39.892733 3314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:39.892895 3314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:39.900493 3314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:39.900682 3314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:39.900810 3314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:39.900925 3314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:39.901012 3314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:06:39.911449 3314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:39.912917 3314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:39.914395 3314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:39.927340 3322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:39.927382 3320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:39.927273 3319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:39.928283 3314 server_base.cc:1034] running on GCE node
I20260619 01:06:39.929353 3314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:39.931097 3314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:39.932348 3314 hybrid_clock.cc:648] HybridClock initialized: now 1781831199932293 us; error 47 us; skew 500 ppm
I20260619 01:06:39.932765 3314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:39 dist-test-slave-5p8w krb5kdc[3301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:39.936259 3314 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:06:39.938095 3314 webserver.cc:460] Webserver started at http://127.0.97.62:39845/ using document root <none> and password file <none>
I20260619 01:06:39.938934 3314 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:39.939098 3314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:39.939446 3314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:39.942912 3314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "7cfae180635d45f285b75671dd36a684"
format_stamp: "Formatted at 2026-06-19 01:06:39 on dist-test-slave-5p8w"
server_key: "06e84511c04bc2148bce82a5b29f37b6"
server_key_iv: "b49f4599268214d8e8edafa4875c0dbc"
server_key_version: "encryptionkey@0"
I20260619 01:06:39.943825 3314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "7cfae180635d45f285b75671dd36a684"
format_stamp: "Formatted at 2026-06-19 01:06:39 on dist-test-slave-5p8w"
server_key: "06e84511c04bc2148bce82a5b29f37b6"
server_key_iv: "b49f4599268214d8e8edafa4875c0dbc"
server_key_version: "encryptionkey@0"
I20260619 01:06:39.950035 3314 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 01:06:39.954319 3329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:39.956008 3314 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:06:39.956274 3314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "7cfae180635d45f285b75671dd36a684"
format_stamp: "Formatted at 2026-06-19 01:06:39 on dist-test-slave-5p8w"
server_key: "06e84511c04bc2148bce82a5b29f37b6"
server_key_iv: "b49f4599268214d8e8edafa4875c0dbc"
server_key_version: "encryptionkey@0"
I20260619 01:06:39.956527 3314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:39.970903 3314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:39.976414 3314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:39.976771 3314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:39.998292 3314 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34573
I20260619 01:06:39.998327 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34573 every 8 connection(s)
I20260619 01:06:39.999956 3314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:40.002730 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 3314
I20260619 01:06:40.002980 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:40.003641 388 external_mini_cluster.cc:1428] Setting key 2cc26f3bea61e83ea1e4a88f98b51d9c
I20260619 01:06:40.004730 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:06:40 dist-test-slave-5p8w krb5kdc[3301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:40.020256 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684: Bootstrap starting.
I20260619 01:06:40.025436 3381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:40.026822 3381 log.cc:824] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:40.030764 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684: No bootstrap required, opened a new log
I20260619 01:06:40.031025 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:40.005822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.0.97.62:34573)
0619 01:06:40.006736 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:40.006754 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:40.006810 (+ 56us) server_negotiation.cc:407] Connection header received
0619 01:06:40.008770 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:40.008806 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:40.009544 (+ 738us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:40.010099 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:40.011649 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:40.013883 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:40.015349 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:40.016515 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:40.016760 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:40.021234 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:40.021246 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:40.021270 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:40.021380 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:40.025819 (+ 4439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:40.026709 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:40.026718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:40.026728 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:40.027039 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:40.027441 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:40.027450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:40.027456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:40.028174 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:40.028517 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:40.028986 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 01:06:40.029507 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":370,"threads_started":1}
I20260619 01:06:40.035605 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:06:40.039330 3381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfae180635d45f285b75671dd36a684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34573 } }
I20260619 01:06:40.039731 3381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:40.039870 3381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfae180635d45f285b75671dd36a684, State: Initialized, Role: FOLLOWER
I20260619 01:06:40.040535 3381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfae180635d45f285b75671dd36a684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34573 } }
I20260619 01:06:40.040795 3381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:40.040942 3381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:40.041129 3381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:40.043469 3381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfae180635d45f285b75671dd36a684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34573 } }
I20260619 01:06:40.044077 3381 leader_election.cc:302] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cfae180635d45f285b75671dd36a684; no voters:
I20260619 01:06:40.044867 3381 leader_election.cc:288] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:40.044970 3387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:40.046231 3387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [term 1 LEADER]: Becoming Leader. State: Replica: 7cfae180635d45f285b75671dd36a684, State: Running, Role: LEADER
I20260619 01:06:40.046923 3387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfae180635d45f285b75671dd36a684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34573 } }
I20260619 01:06:40.047505 3381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:40.050235 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cfae180635d45f285b75671dd36a684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfae180635d45f285b75671dd36a684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34573 } } }
I20260619 01:06:40.050706 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:40.053565 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cfae180635d45f285b75671dd36a684. Latest consensus state: current_term: 1 leader_uuid: "7cfae180635d45f285b75671dd36a684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cfae180635d45f285b75671dd36a684" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34573 } } }
I20260619 01:06:40.054296 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:40.056638 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:40.063190 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:40.069998 3405 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 01:06:40.076975 3392 catalog_manager.cc:1347] Generated new cluster ID: 40c929af684640beae4ff3347e1a0ea8
I20260619 01:06:40.077227 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:40.087001 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:40.078725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.0.97.62:34573)
0619 01:06:40.079172 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:40.079180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:40.079546 (+ 366us) server_negotiation.cc:407] Connection header received
0619 01:06:40.079761 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:40.079766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:40.079849 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:40.079967 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:40.080986 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:40.081894 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:40.083209 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:40.083667 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:40.083831 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:40.084988 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:40.085002 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:40.085006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:40.085824 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:40.085882 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:40.085935 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:40.086048 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:40.086255 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 01:06:40.086359 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287}
I20260619 01:06:40.088459 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:40.090489 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:40.099910 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cfae180635d45f285b75671dd36a684: Generated new TSK 0
I20260619 01:06:40.101133 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:06:40.117257 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3314
I20260619 01:06:40.118839 3376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:40.210240 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 3314
2026-06-19T01:06:40Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1975 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:06:40 dist-test-slave-5p8w krb5kdc[3415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:06:40 dist-test-slave-5p8w krb5kdc[3415](info): set up 2 sockets
Jun 19 01:06:40 dist-test-slave-5p8w krb5kdc[3415](info): commencing operation
krb5kdc: starting...
W20260619 01:06:41.313823 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:06:41 dist-test-slave-5p8w krb5kdc[3415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:06:41.525182 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44211
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:41.871404 3428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:41.871742 3428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:41.871838 3428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:41.879359 3428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:41.879491 3428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:41.879539 3428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:41.879590 3428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:41.879635 3428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:06:41.890285 3428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:41.891659 3428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:41.892963 3428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:41.905606 3436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:41.905635 3434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:41.905702 3433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:41.911520 3428 server_base.cc:1034] running on GCE node
I20260619 01:06:41.912506 3428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:41.914270 3428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:41.915501 3428 hybrid_clock.cc:648] HybridClock initialized: now 1781831201915451 us; error 40 us; skew 500 ppm
I20260619 01:06:41.915894 3428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:41 dist-test-slave-5p8w krb5kdc[3415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:41.919382 3428 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:06:41.921214 3428 webserver.cc:460] Webserver started at http://127.0.97.62:37711/ using document root <none> and password file <none>
I20260619 01:06:41.922137 3428 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:41.922282 3428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:41.922639 3428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:41.926173 3428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "5be2061efd644450ba0332551d7cc676"
format_stamp: "Formatted at 2026-06-19 01:06:41 on dist-test-slave-5p8w"
server_key: "b316f5fc2a4a17d5c7e5c7f0265dac04"
server_key_iv: "3e77d71e15d6e558595e6eed38af3fdc"
server_key_version: "encryptionkey@0"
I20260619 01:06:41.927073 3428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "5be2061efd644450ba0332551d7cc676"
format_stamp: "Formatted at 2026-06-19 01:06:41 on dist-test-slave-5p8w"
server_key: "b316f5fc2a4a17d5c7e5c7f0265dac04"
server_key_iv: "3e77d71e15d6e558595e6eed38af3fdc"
server_key_version: "encryptionkey@0"
I20260619 01:06:41.934167 3428 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260619 01:06:41.939196 3443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:41.941717 3428 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260619 01:06:41.941998 3428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "5be2061efd644450ba0332551d7cc676"
format_stamp: "Formatted at 2026-06-19 01:06:41 on dist-test-slave-5p8w"
server_key: "b316f5fc2a4a17d5c7e5c7f0265dac04"
server_key_iv: "3e77d71e15d6e558595e6eed38af3fdc"
server_key_version: "encryptionkey@0"
I20260619 01:06:41.942257 3428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:41.959532 3428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:41.964231 3428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:41.964547 3428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:41.984251 3428 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44211
I20260619 01:06:41.984272 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44211 every 8 connection(s)
I20260619 01:06:41.985843 3428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:41.986390 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 3428
I20260619 01:06:41.986655 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:41.987449 388 external_mini_cluster.cc:1428] Setting key 993cdfd600603dffedcfedda0c77862e
I20260619 01:06:41.990877 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:06:42 dist-test-slave-5p8w krb5kdc[3415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:42.005100 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676: Bootstrap starting.
I20260619 01:06:42.009836 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:42.011165 3495 log.cc:824] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:42.014580 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:41.990202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.0.97.62:44211)
0619 01:06:41.991025 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:41.991042 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:41.991096 (+ 54us) server_negotiation.cc:407] Connection header received
0619 01:06:41.993436 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:41.993466 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:41.994052 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:41.994599 (+ 547us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:41.995972 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:41.998217 (+ 2245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:41.999596 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:42.000661 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:42.000872 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:42.005426 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:42.005439 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:42.005465 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:42.005534 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:42.009648 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:42.010534 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:42.010542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:42.010552 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:42.010787 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:42.011353 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:42.011361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:42.011365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:42.012172 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:42.012534 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:42.013041 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 01:06:42.013544 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":333,"threads_started":1}
I20260619 01:06:42.014694 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676: No bootstrap required, opened a new log
I20260619 01:06:42.023095 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be2061efd644450ba0332551d7cc676" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44211 } }
I20260619 01:06:42.023571 3495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:42.023671 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be2061efd644450ba0332551d7cc676, State: Initialized, Role: FOLLOWER
I20260619 01:06:42.024253 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be2061efd644450ba0332551d7cc676" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44211 } }
I20260619 01:06:42.024466 3495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:42.024598 3495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:42.024789 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:42.027352 3495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be2061efd644450ba0332551d7cc676" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44211 } }
I20260619 01:06:42.027886 3495 leader_election.cc:302] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5be2061efd644450ba0332551d7cc676; no voters:
I20260619 01:06:42.028455 3495 leader_election.cc:288] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:42.028621 3500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:42.029418 3500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [term 1 LEADER]: Becoming Leader. State: Replica: 5be2061efd644450ba0332551d7cc676, State: Running, Role: LEADER
I20260619 01:06:42.029986 3500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be2061efd644450ba0332551d7cc676" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44211 } }
I20260619 01:06:42.030898 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:42.033176 3502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5be2061efd644450ba0332551d7cc676. Latest consensus state: current_term: 1 leader_uuid: "5be2061efd644450ba0332551d7cc676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be2061efd644450ba0332551d7cc676" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44211 } } }
I20260619 01:06:42.033337 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5be2061efd644450ba0332551d7cc676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be2061efd644450ba0332551d7cc676" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44211 } } }
I20260619 01:06:42.033635 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:42.033644 3502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:42.034567 3508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:42.045387 3508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:42.055922 3508 catalog_manager.cc:1347] Generated new cluster ID: 46c116b85acc4b39be8fa450fa8c600f
I20260619 01:06:42.056075 3508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:42.071602 3508 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:42.073112 3508 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:42.084877 3508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5be2061efd644450ba0332551d7cc676: Generated new TSK 0
I20260619 01:06:42.085933 3508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:06:42.140588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44211
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:42.495923 3521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:42.496260 3521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:42.496357 3521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:42.503960 3521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:42.504187 3521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:42.504376 3521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:06:42.515280 3521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:42.516669 3521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:42.518116 3521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:42.532796 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:42.532747 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:42.532766 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:42.534474 3521 server_base.cc:1034] running on GCE node
I20260619 01:06:42.535216 3521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:42.536271 3521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:42.537516 3521 hybrid_clock.cc:648] HybridClock initialized: now 1781831202537452 us; error 60 us; skew 500 ppm
I20260619 01:06:42.537887 3521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:42 dist-test-slave-5p8w krb5kdc[3415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:42.546166 3521 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:06:42.548103 3521 webserver.cc:460] Webserver started at http://127.0.97.1:38911/ using document root <none> and password file <none>
I20260619 01:06:42.548981 3521 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:42.549160 3521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:42.549540 3521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:42.552737 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "df50068660e949e2aa520b0e4504539a"
format_stamp: "Formatted at 2026-06-19 01:06:42 on dist-test-slave-5p8w"
server_key: "625d7b824d63e8befdc19ce6193ade42"
server_key_iv: "8e9577cfbfce0025cc716723209a094a"
server_key_version: "encryptionkey@0"
I20260619 01:06:42.553769 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "df50068660e949e2aa520b0e4504539a"
format_stamp: "Formatted at 2026-06-19 01:06:42 on dist-test-slave-5p8w"
server_key: "625d7b824d63e8befdc19ce6193ade42"
server_key_iv: "8e9577cfbfce0025cc716723209a094a"
server_key_version: "encryptionkey@0"
I20260619 01:06:42.559809 3521 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 01:06:42.563809 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:42.565450 3521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:42.565655 3521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "df50068660e949e2aa520b0e4504539a"
format_stamp: "Formatted at 2026-06-19 01:06:42 on dist-test-slave-5p8w"
server_key: "625d7b824d63e8befdc19ce6193ade42"
server_key_iv: "8e9577cfbfce0025cc716723209a094a"
server_key_version: "encryptionkey@0"
I20260619 01:06:42.565878 3521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:42.592093 3521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:42.597105 3521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:42.597497 3521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:42.598599 3521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:42.600119 3521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:42.600230 3521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:42.600386 3521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:42.600457 3521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:42.635525 3521 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45073
I20260619 01:06:42.635617 3648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45073 every 8 connection(s)
I20260619 01:06:42.636979 3521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:06:42.642635 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 3521
I20260619 01:06:42.642858 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:06:42.643620 388 external_mini_cluster.cc:1428] Setting key 487751a86749c294d7ebb6cc3310f468
Jun 19 01:06:42 dist-test-slave-5p8w krb5kdc[3415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:42.660909 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:42.639638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56175 (local address 127.0.97.62:44211)
0619 01:06:42.640093 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:42.640098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:42.641012 (+ 914us) server_negotiation.cc:407] Connection header received
0619 01:06:42.643089 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:42.643096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:42.643197 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:42.643322 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:42.645208 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:42.646313 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:42.648195 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:42.648958 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:42.649114 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:42.653146 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:42.653156 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:42.653165 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:42.653214 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:42.656945 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:42.657907 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:42.657916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:42.657923 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:42.658080 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:42.658532 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:42.658541 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:42.658547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:42.658973 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:42.659140 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:42.660090 (+ 950us) server_negotiation.cc:299] Negotiation successful
0619 01:06:42.660383 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":254,"threads_started":1}
I20260619 01:06:42.661669 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:42.640110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44211 (local address 127.0.97.1:56175)
0619 01:06:42.640839 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:42.640884 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:42.642668 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:42.643546 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:42.643555 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:42.644180 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:42.645029 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:42.645057 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:42.646485 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:42.646489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:42.648033 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:42.648047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:42.649231 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:42.649238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:42.649576 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:42.650396 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:42.650424 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:42.652863 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:42.657155 (+ 4292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:42.657163 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:42.657180 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:42.657748 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:42.658197 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:42.658199 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:42.658202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:42.658399 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:42.659101 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:42.659108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:42.659613 (+ 505us) client_negotiation.cc:769] Sending connection context
0619 01:06:42.659926 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 01:06:42.660378 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":271,"threads_started":1}
I20260619 01:06:42.663476 3649 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44211
I20260619 01:06:42.663887 3649 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:06:42.664816 3649 heartbeater.cc:511] Master 127.0.97.62:44211 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:06:42.667392 3460 ts_manager.cc:194] Registered new tserver with Master: df50068660e949e2aa520b0e4504539a (127.0.97.1:45073)
I20260619 01:06:42.669865 3460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:06:42.692549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44211
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:43.030553 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:43.030874 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:43.031040 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:43.038977 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:43.039115 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:43.039287 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:06:43.049975 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:43.051415 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:43.052824 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:43.066941 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:43.066896 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:43.067412 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:43.067886 3655 server_base.cc:1034] running on GCE node
I20260619 01:06:43.068529 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:43.069794 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:43.071044 3655 hybrid_clock.cc:648] HybridClock initialized: now 1781831203070984 us; error 58 us; skew 500 ppm
I20260619 01:06:43.071408 3655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:43 dist-test-slave-5p8w krb5kdc[3415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:43.074731 3655 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:06:43.076300 3655 webserver.cc:460] Webserver started at http://127.0.97.2:44647/ using document root <none> and password file <none>
I20260619 01:06:43.077087 3655 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:43.077237 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:43.077589 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:43.080730 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "369f2cfaf879483787856d77c4e02837"
format_stamp: "Formatted at 2026-06-19 01:06:43 on dist-test-slave-5p8w"
server_key: "a6281c53b8ec54d1a9ba81372bbb0b1a"
server_key_iv: "9fa858aece2acd81525d06fbf642b870"
server_key_version: "encryptionkey@0"
I20260619 01:06:43.081787 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "369f2cfaf879483787856d77c4e02837"
format_stamp: "Formatted at 2026-06-19 01:06:43 on dist-test-slave-5p8w"
server_key: "a6281c53b8ec54d1a9ba81372bbb0b1a"
server_key_iv: "9fa858aece2acd81525d06fbf642b870"
server_key_version: "encryptionkey@0"
I20260619 01:06:43.087447 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:06:43.091463 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:43.093030 3655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:06:43.093230 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "369f2cfaf879483787856d77c4e02837"
format_stamp: "Formatted at 2026-06-19 01:06:43 on dist-test-slave-5p8w"
server_key: "a6281c53b8ec54d1a9ba81372bbb0b1a"
server_key_iv: "9fa858aece2acd81525d06fbf642b870"
server_key_version: "encryptionkey@0"
I20260619 01:06:43.093533 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:43.107887 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:43.112556 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:43.112880 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:43.113986 3655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:43.115537 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:43.115639 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:43.115789 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:43.115865 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:43.157457 3655 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40699
I20260619 01:06:43.157546 3782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40699 every 8 connection(s)
I20260619 01:06:43.158797 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:06:43.162832 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 3655
I20260619 01:06:43.163084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:06:43.163542 388 external_mini_cluster.cc:1428] Setting key 8c02367992c67efb8390ab1d01912130
Jun 19 01:06:43 dist-test-slave-5p8w krb5kdc[3415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:06:43.183558 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:43.161907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56571 (local address 127.0.97.62:44211)
0619 01:06:43.162485 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:43.162494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:43.163617 (+ 1123us) server_negotiation.cc:407] Connection header received
0619 01:06:43.165826 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:43.165841 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:43.165939 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:43.166081 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:43.167714 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.169029 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.170917 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.171711 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.171881 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:43.176487 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:43.176492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:43.176496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:43.176535 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:43.179639 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.180851 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.180856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.180859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.181006 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.181648 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.181652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.181654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.181922 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:43.182035 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:43.183073 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0619 01:06:43.183287 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":326,"threads_started":1}
I20260619 01:06:43.184762 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:43.162569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44211 (local address 127.0.97.2:56571)
0619 01:06:43.163411 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:43.163465 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:43.165446 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:43.166299 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:43.166313 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:43.166871 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:43.167499 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:43.167519 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.169207 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.169211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:43.170704 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:43.170726 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.172021 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.172031 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:43.172310 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:43.173140 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:43.173166 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:43.176157 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:43.179856 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:43.179871 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:43.179895 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:43.180655 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:43.181144 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:43.181151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:43.181156 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:43.181511 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:43.182065 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:43.182076 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:43.182615 (+ 539us) client_negotiation.cc:769] Sending connection context
0619 01:06:43.182947 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 01:06:43.183533 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":353,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 01:06:43.186535 3783 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44211
I20260619 01:06:43.186913 3783 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:43.187719 3783 heartbeater.cc:511] Master 127.0.97.62:44211 requested a full tablet report, sending...
I20260619 01:06:43.189450 3460 ts_manager.cc:194] Registered new tserver with Master: 369f2cfaf879483787856d77c4e02837 (127.0.97.2:40699)
I20260619 01:06:43.190763 3460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:06:43.211911 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44211
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:43.547712 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:43.548077 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:43.548259 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:43.555770 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:43.555950 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:43.556141 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:06:43.566282 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:43.567711 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:43.569155 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:43.583283 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:43.583307 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:43.584139 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:43.584152 3789 server_base.cc:1034] running on GCE node
I20260619 01:06:43.584905 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:43.585966 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:43.587193 3789 hybrid_clock.cc:648] HybridClock initialized: now 1781831203587148 us; error 45 us; skew 500 ppm
I20260619 01:06:43.587525 3789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:43 dist-test-slave-5p8w krb5kdc[3415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:43.591094 3789 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:06:43.592553 3789 webserver.cc:460] Webserver started at http://127.0.97.3:32789/ using document root <none> and password file <none>
I20260619 01:06:43.593263 3789 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:43.593437 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:43.593708 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:43.596669 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "b05536515f6243048d496e4fcc1e744c"
format_stamp: "Formatted at 2026-06-19 01:06:43 on dist-test-slave-5p8w"
server_key: "714583289692364551faa8d35fd0d665"
server_key_iv: "f03ad28b092f90c101ac4d143d290431"
server_key_version: "encryptionkey@0"
I20260619 01:06:43.597509 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b05536515f6243048d496e4fcc1e744c"
format_stamp: "Formatted at 2026-06-19 01:06:43 on dist-test-slave-5p8w"
server_key: "714583289692364551faa8d35fd0d665"
server_key_iv: "f03ad28b092f90c101ac4d143d290431"
server_key_version: "encryptionkey@0"
I20260619 01:06:43.603068 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:06:43.606755 3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:43.608220 3789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:06:43.608441 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "b05536515f6243048d496e4fcc1e744c"
format_stamp: "Formatted at 2026-06-19 01:06:43 on dist-test-slave-5p8w"
server_key: "714583289692364551faa8d35fd0d665"
server_key_iv: "f03ad28b092f90c101ac4d143d290431"
server_key_version: "encryptionkey@0"
I20260619 01:06:43.608657 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:43.619822 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:43.624082 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:43.624413 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:43.625463 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:43.626814 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:43.626924 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:43.627081 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:43.627154 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:43.661861 3789 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34673
I20260619 01:06:43.661949 3916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34673 every 8 connection(s)
I20260619 01:06:43.663218 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:06:43.671001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 3789
I20260619 01:06:43.671211 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:06:43.671658 388 external_mini_cluster.cc:1428] Setting key 5b6fa902bcb81c6f7bd082f975fafc4f
I20260619 01:06:43.673528 3649 heartbeater.cc:503] Master 127.0.97.62:44211 was elected leader, sending a full tablet report...
Jun 19 01:06:43 dist-test-slave-5p8w krb5kdc[3415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:43.683770 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:43.665928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58741 (local address 127.0.97.62:44211)
0619 01:06:43.666101 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:43.666105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:43.667320 (+ 1215us) server_negotiation.cc:407] Connection header received
0619 01:06:43.669135 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:43.669141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:43.669261 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:43.669407 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:43.670961 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.671829 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.673804 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.674450 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.674581 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:43.678202 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:43.678207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:43.678210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:43.678243 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:43.680614 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.681462 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.681467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.681469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.681580 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.682023 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.682028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.682031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.682306 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:43.682462 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:43.683187 (+ 725us) server_negotiation.cc:299] Negotiation successful
0619 01:06:43.683417 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:06:43.684782 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:43.666465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44211 (local address 127.0.97.3:58741)
0619 01:06:43.667151 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:43.667188 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:43.668829 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:43.669642 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:43.669650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:43.670154 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:43.670798 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:43.670816 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.671972 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.671980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:43.673639 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:43.673658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.674865 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.674870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:43.675065 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:43.675682 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:43.675704 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:43.677978 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:43.680785 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:43.680795 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:43.680814 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:43.681284 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:43.681695 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:43.681701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:43.681705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:43.681900 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:43.682437 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:43.682446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:43.682817 (+ 371us) client_negotiation.cc:769] Sending connection context
0619 01:06:43.683085 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 01:06:43.683655 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":271,"threads_started":1}
I20260619 01:06:43.686264 3917 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44211
I20260619 01:06:43.686612 3917 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:43.687494 3917 heartbeater.cc:511] Master 127.0.97.62:44211 requested a full tablet report, sending...
I20260619 01:06:43.688867 3460 ts_manager.cc:194] Registered new tserver with Master: b05536515f6243048d496e4fcc1e744c (127.0.97.3:34673)
I20260619 01:06:43.689715 3460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58741
I20260619 01:06:43.700088 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 01:06:43 dist-test-slave-5p8w krb5kdc[3415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831203, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 01:06:43 dist-test-slave-5p8w krb5kdc[3415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831203, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:43.749801 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:43.735396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.0.97.62:44211)
0619 01:06:43.735644 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:43.735653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:43.736054 (+ 401us) server_negotiation.cc:407] Connection header received
0619 01:06:43.736201 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:43.736204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:43.736286 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:43.736375 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:43.737752 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.738479 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.739775 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.740190 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.740302 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:43.744451 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:43.744457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:43.744461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:43.744500 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:43.746999 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.747691 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.747697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.747702 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.747849 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.748279 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.748288 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.748291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.748599 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:43.748912 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:43.749174 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 01:06:43.749430 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 01:06:43.750422 3460 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36010
I20260619 01:06:43.752305 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3521
Jun 19 01:06:43 dist-test-slave-5p8w krb5kdc[3415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831203, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:06:43.767333 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:43.753224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.0.97.1:45073)
0619 01:06:43.753879 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:43.753886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:43.753912 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:06:43.753971 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:43.753977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:43.754184 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:43.754331 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:43.755432 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.756428 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.757649 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.758220 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.758326 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:43.762172 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:43.762186 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:43.762201 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:43.762247 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:43.764926 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.765623 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.765631 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.765634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.765760 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.766069 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.766072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.766074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.766408 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:43.766621 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:43.766834 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 01:06:43.767025 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":364,"threads_started":1}
W20260619 01:06:43.768627 3644 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57016
W20260619 01:06:43.769918 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3521: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:06:43.770128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 3521
I20260619 01:06:43.786353 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3655
Jun 19 01:06:43 dist-test-slave-5p8w krb5kdc[3415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831203, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:06:43.803046 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:43.787250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.0.97.2:40699)
0619 01:06:43.787747 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:43.787754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:43.787778 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:06:43.787836 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:43.787842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:43.788055 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:43.788225 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:43.789553 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.790554 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.791860 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.792425 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.792571 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:43.796948 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:43.796963 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:43.796979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:43.797019 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:43.799911 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.800657 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.800664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.800667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.800805 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.801196 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.801203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.801206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.801715 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:43.802050 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:43.802371 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 01:06:43.802665 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":243,"threads_started":1}
W20260619 01:06:43.804445 3778 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35238
W20260619 01:06:43.805658 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3655: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:06:43.805881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 3655
I20260619 01:06:43.821777 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3789
Jun 19 01:06:43 dist-test-slave-5p8w krb5kdc[3415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831203, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:06:43.838833 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:43.822739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.0.97.3:34673)
0619 01:06:43.823322 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:43.823330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:43.823360 (+ 30us) server_negotiation.cc:407] Connection header received
0619 01:06:43.823420 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:43.823426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:43.823640 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:43.823790 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:43.825149 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.826262 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.827516 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:43.828103 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:43.828205 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:43.832874 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:43.832891 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:43.832908 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:43.832959 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:43.835899 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.836811 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.836819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.836822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.836973 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:43.837382 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:43.837387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:43.837390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:43.837811 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:43.838054 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:43.838309 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 01:06:43.838526 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":281,"threads_started":1}
W20260619 01:06:43.840216 3912 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38290
W20260619 01:06:43.841459 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3789: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:06:43.841631 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 3789
I20260619 01:06:43.857717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3428
I20260619 01:06:43.858960 3490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:43.961337 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 3428
2026-06-19T01:06:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3756 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:06:44 dist-test-slave-5p8w krb5kdc[3934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:06:44 dist-test-slave-5p8w krb5kdc[3934](info): set up 2 sockets
Jun 19 01:06:44 dist-test-slave-5p8w krb5kdc[3934](info): commencing operation
krb5kdc: starting...
W20260619 01:06:45.069734 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:06:45 dist-test-slave-5p8w krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:06:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:06:45.283107 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34003
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:45.623301 3947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:45.623675 3947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:45.623843 3947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:45.631383 3947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:45.631587 3947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:45.631659 3947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:45.631728 3947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:45.631773 3947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:06:45.642073 3947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:45.643400 3947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:45.644694 3947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:45.657975 3952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:45.657852 3955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:45.657946 3953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:45.659154 3947 server_base.cc:1034] running on GCE node
I20260619 01:06:45.660056 3947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:45.661682 3947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:45.662911 3947 hybrid_clock.cc:648] HybridClock initialized: now 1781831205662844 us; error 61 us; skew 500 ppm
I20260619 01:06:45.663326 3947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:45 dist-test-slave-5p8w krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:45.666783 3947 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:06:45.668391 3947 webserver.cc:460] Webserver started at http://127.0.97.62:37871/ using document root <none> and password file <none>
I20260619 01:06:45.669133 3947 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:45.669241 3947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:45.669600 3947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:45.672755 3947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "2a3981ba3dd243fea537e914631347ba"
format_stamp: "Formatted at 2026-06-19 01:06:45 on dist-test-slave-5p8w"
server_key: "176b65761da6e50cfca5399a036c0318"
server_key_iv: "1ad28fbc7851e3aee44f63ffbe5112a3"
server_key_version: "encryptionkey@0"
I20260619 01:06:45.673609 3947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "2a3981ba3dd243fea537e914631347ba"
format_stamp: "Formatted at 2026-06-19 01:06:45 on dist-test-slave-5p8w"
server_key: "176b65761da6e50cfca5399a036c0318"
server_key_iv: "1ad28fbc7851e3aee44f63ffbe5112a3"
server_key_version: "encryptionkey@0"
I20260619 01:06:45.679219 3947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:06:45.683194 3962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:45.684733 3947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:06:45.684892 3947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "2a3981ba3dd243fea537e914631347ba"
format_stamp: "Formatted at 2026-06-19 01:06:45 on dist-test-slave-5p8w"
server_key: "176b65761da6e50cfca5399a036c0318"
server_key_iv: "1ad28fbc7851e3aee44f63ffbe5112a3"
server_key_version: "encryptionkey@0"
I20260619 01:06:45.685081 3947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:45.700533 3947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:45.705092 3947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:45.705447 3947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:45.724938 3947 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34003
I20260619 01:06:45.724992 4013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34003 every 8 connection(s)
I20260619 01:06:45.726547 3947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:45.730522 4014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:45.733887 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 3947
I20260619 01:06:45.734098 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:45.734874 388 external_mini_cluster.cc:1428] Setting key 3d414f5c378ccf26d68f13b029462932
I20260619 01:06:45.744053 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba: Bootstrap starting.
I20260619 01:06:45.747427 4014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:45.748910 4014 log.cc:824] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba: Log is configured to *not* fsync() on all Append() calls
Jun 19 01:06:45 dist-test-slave-5p8w krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:45.752277 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba: No bootstrap required, opened a new log
I20260619 01:06:45.760046 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:45.737364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.0.97.62:34003)
0619 01:06:45.738436 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:45.738450 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:45.738498 (+ 48us) server_negotiation.cc:407] Connection header received
0619 01:06:45.740312 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:45.740341 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:45.740875 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:45.741419 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:45.743034 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:45.744487 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:45.745922 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:45.746640 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:45.746857 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:45.751757 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:45.751766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:45.751784 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:45.751823 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:45.755270 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:45.756124 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:45.756132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:45.756141 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:45.756338 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:45.756780 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:45.756783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:45.756785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:45.757522 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:45.757896 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:45.758398 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 01:06:45.758879 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":524,"threads_started":1}
I20260619 01:06:45.760666 4014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3981ba3dd243fea537e914631347ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34003 } }
I20260619 01:06:45.761153 4014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:45.761262 4014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3981ba3dd243fea537e914631347ba, State: Initialized, Role: FOLLOWER
I20260619 01:06:45.761966 4014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3981ba3dd243fea537e914631347ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34003 } }
I20260619 01:06:45.762216 4014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:45.762374 4014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:45.762566 4014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:45.765089 4014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3981ba3dd243fea537e914631347ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34003 } }
I20260619 01:06:45.765760 4014 leader_election.cc:302] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a3981ba3dd243fea537e914631347ba; no voters:
I20260619 01:06:45.766429 4014 leader_election.cc:288] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:45.766583 4019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:45.767340 4019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [term 1 LEADER]: Becoming Leader. State: Replica: 2a3981ba3dd243fea537e914631347ba, State: Running, Role: LEADER
I20260619 01:06:45.767941 4019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3981ba3dd243fea537e914631347ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34003 } }
I20260619 01:06:45.768564 4014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:45.770823 4020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a3981ba3dd243fea537e914631347ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3981ba3dd243fea537e914631347ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34003 } } }
I20260619 01:06:45.771191 4020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:45.771776 4021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a3981ba3dd243fea537e914631347ba. Latest consensus state: current_term: 1 leader_uuid: "2a3981ba3dd243fea537e914631347ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3981ba3dd243fea537e914631347ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34003 } } }
I20260619 01:06:45.772143 4024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:45.772217 4021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:45.778314 4024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:45.788753 4024 catalog_manager.cc:1347] Generated new cluster ID: d1e40205bb3042b3ad7d9936b7e1ba17
I20260619 01:06:45.788882 4024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:45.824862 4024 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:45.826355 4024 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:45.836062 4024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a3981ba3dd243fea537e914631347ba: Generated new TSK 0
I20260619 01:06:45.837001 4024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:06:45.889513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34003
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:46.247696 4040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:46.248103 4040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:46.248243 4040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:46.256139 4040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:46.256283 4040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:46.256404 4040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:06:46.267104 4040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:46.268771 4040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:46.270447 4040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:46.285558 4045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:46.285570 4046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:46.285614 4048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:46.286597 4040 server_base.cc:1034] running on GCE node
I20260619 01:06:46.287269 4040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:46.288288 4040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:46.289496 4040 hybrid_clock.cc:648] HybridClock initialized: now 1781831206289442 us; error 48 us; skew 500 ppm
I20260619 01:06:46.289865 4040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:46 dist-test-slave-5p8w krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:46.293622 4040 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:06:46.295192 4040 webserver.cc:460] Webserver started at http://127.0.97.1:36561/ using document root <none> and password file <none>
I20260619 01:06:46.295931 4040 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:46.296044 4040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:46.296316 4040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:46.300376 4040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "0ff4ff70dad3480ba476c0695a396124"
format_stamp: "Formatted at 2026-06-19 01:06:46 on dist-test-slave-5p8w"
server_key: "4ff5a916565cd66504e62558e5a4e860"
server_key_iv: "ebb8053186d1e28b77ab1332ef7310ac"
server_key_version: "encryptionkey@0"
I20260619 01:06:46.301685 4040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0ff4ff70dad3480ba476c0695a396124"
format_stamp: "Formatted at 2026-06-19 01:06:46 on dist-test-slave-5p8w"
server_key: "4ff5a916565cd66504e62558e5a4e860"
server_key_iv: "ebb8053186d1e28b77ab1332ef7310ac"
server_key_version: "encryptionkey@0"
I20260619 01:06:46.308079 4040 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 01:06:46.312053 4055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:46.313740 4040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 01:06:46.313959 4040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "0ff4ff70dad3480ba476c0695a396124"
format_stamp: "Formatted at 2026-06-19 01:06:46 on dist-test-slave-5p8w"
server_key: "4ff5a916565cd66504e62558e5a4e860"
server_key_iv: "ebb8053186d1e28b77ab1332ef7310ac"
server_key_version: "encryptionkey@0"
I20260619 01:06:46.314208 4040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:46.343947 4040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:46.348850 4040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:46.349190 4040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:46.350248 4040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:46.351709 4040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:46.351820 4040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:46.351981 4040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:46.352059 4040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:46.386787 4040 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38861
I20260619 01:06:46.386826 4167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38861 every 8 connection(s)
I20260619 01:06:46.388216 4040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:06:46.391350 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 4040
I20260619 01:06:46.391556 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:06:46.392385 388 external_mini_cluster.cc:1428] Setting key 65df833c7c76fc4f2ecc0f72cf8ec24a
Jun 19 01:06:46 dist-test-slave-5p8w krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:06:46.412377 4170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:46.391153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41633 (local address 127.0.97.62:34003)
0619 01:06:46.391659 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:46.391665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:46.393212 (+ 1547us) server_negotiation.cc:407] Connection header received
0619 01:06:46.395461 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:46.395469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:46.395562 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:46.395690 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:46.397394 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:46.398581 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:46.400730 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:46.401130 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:46.401249 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:46.405918 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:46.405924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:46.405928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:46.405967 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:46.408939 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:46.409942 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:46.409949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:46.409953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:46.410104 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:46.410577 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:46.410585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:46.410593 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:46.410864 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:46.411032 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:46.411767 (+ 735us) server_negotiation.cc:299] Negotiation successful
0619 01:06:46.412008 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":252,"threads_started":1}
I20260619 01:06:46.413561 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:46.392033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34003 (local address 127.0.97.1:41633)
0619 01:06:46.392975 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:46.393013 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:46.395164 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:46.395944 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:46.395953 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:46.396552 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:46.397161 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:46.397179 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:46.398756 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:46.398760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:46.400535 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:46.400560 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:46.401426 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:46.401436 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:46.401814 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:46.402615 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:46.402653 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:46.405621 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:46.409198 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:46.409215 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:46.409232 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:46.409793 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:46.410227 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:46.410232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:46.410235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:46.410449 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:46.410999 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:46.411009 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:46.411375 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 01:06:46.411660 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 01:06:46.412240 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":423,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:06:46.415284 4168 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34003
I20260619 01:06:46.415627 4168 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:46.416589 4168 heartbeater.cc:511] Master 127.0.97.62:34003 requested a full tablet report, sending...
I20260619 01:06:46.419287 3979 ts_manager.cc:194] Registered new tserver with Master: 0ff4ff70dad3480ba476c0695a396124 (127.0.97.1:38861)
I20260619 01:06:46.421257 3979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:06:46.441803 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34003
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:46.778954 4174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:46.779263 4174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:46.779448 4174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:46.786948 4174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:46.787073 4174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:46.787252 4174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:06:46.797938 4174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:46.799376 4174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:46.800794 4174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:46.814991 4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:46.815055 4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:46.815032 4179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:46.815832 4174 server_base.cc:1034] running on GCE node
I20260619 01:06:46.816488 4174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:46.817531 4174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:46.818750 4174 hybrid_clock.cc:648] HybridClock initialized: now 1781831206818698 us; error 44 us; skew 500 ppm
I20260619 01:06:46.819161 4174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:46 dist-test-slave-5p8w krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:46.822557 4174 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:06:46.824193 4174 webserver.cc:460] Webserver started at http://127.0.97.2:38565/ using document root <none> and password file <none>
I20260619 01:06:46.824963 4174 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:46.825131 4174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:46.825486 4174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:46.828514 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "c6ddcbf8768849b4bd70912eb5f4e01f"
format_stamp: "Formatted at 2026-06-19 01:06:46 on dist-test-slave-5p8w"
server_key: "a6dace06ed0894d7ea27d500d99d0f31"
server_key_iv: "33cb146d60c597442b5ec48fb2f69d99"
server_key_version: "encryptionkey@0"
I20260619 01:06:46.829401 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c6ddcbf8768849b4bd70912eb5f4e01f"
format_stamp: "Formatted at 2026-06-19 01:06:46 on dist-test-slave-5p8w"
server_key: "a6dace06ed0894d7ea27d500d99d0f31"
server_key_iv: "33cb146d60c597442b5ec48fb2f69d99"
server_key_version: "encryptionkey@0"
I20260619 01:06:46.835160 4174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:06:46.839053 4189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:46.840752 4174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:06:46.840961 4174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "c6ddcbf8768849b4bd70912eb5f4e01f"
format_stamp: "Formatted at 2026-06-19 01:06:46 on dist-test-slave-5p8w"
server_key: "a6dace06ed0894d7ea27d500d99d0f31"
server_key_iv: "33cb146d60c597442b5ec48fb2f69d99"
server_key_version: "encryptionkey@0"
I20260619 01:06:46.841177 4174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:46.855748 4174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:46.860970 4174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:46.861338 4174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:46.862407 4174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:46.863920 4174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:46.863999 4174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:46.864158 4174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:46.864218 4174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:46.899427 4174 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42617
I20260619 01:06:46.899473 4301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42617 every 8 connection(s)
I20260619 01:06:46.900833 4174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:06:46.901958 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 4174
I20260619 01:06:46.902146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:06:46.902674 388 external_mini_cluster.cc:1428] Setting key 8cf0e42cc722befdc00dff2af3b7251b
Jun 19 01:06:46 dist-test-slave-5p8w krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:06:46.925333 4170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:46.904103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36643 (local address 127.0.97.62:34003)
0619 01:06:46.904289 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:46.904294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:46.905670 (+ 1376us) server_negotiation.cc:407] Connection header received
0619 01:06:46.907595 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:46.907601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:46.907703 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:46.907841 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:46.909642 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:46.910362 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:46.911982 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:46.912752 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:46.912903 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:46.917528 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:46.917538 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:46.917547 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:46.917598 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:46.921604 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:46.922574 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:46.922581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:46.922585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:46.922751 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:46.923238 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:46.923243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:46.923246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:46.923670 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:46.923821 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:46.924645 (+ 824us) server_negotiation.cc:299] Negotiation successful
0619 01:06:46.924829 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:06:46.926354 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:46.904669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34003 (local address 127.0.97.2:36643)
0619 01:06:46.905507 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:46.905557 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:46.907289 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:46.908117 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:46.908126 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:46.908729 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:46.909445 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:46.909476 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:46.910537 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:46.910541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:46.911822 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:46.911835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:46.913081 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:46.913088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:46.913516 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:46.914332 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:46.914356 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:46.917141 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:46.921833 (+ 4692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:46.921842 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:46.921859 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:46.922396 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:46.922881 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:46.922885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:46.922887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:46.923096 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:46.923809 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:46.923815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:46.924263 (+ 448us) client_negotiation.cc:769] Sending connection context
0619 01:06:46.924514 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 01:06:46.924934 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":339,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 01:06:46.928717 4302 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34003
I20260619 01:06:46.929116 4302 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:46.929952 4302 heartbeater.cc:511] Master 127.0.97.62:34003 requested a full tablet report, sending...
I20260619 01:06:46.931633 3979 ts_manager.cc:194] Registered new tserver with Master: c6ddcbf8768849b4bd70912eb5f4e01f (127.0.97.2:42617)
I20260619 01:06:46.932556 3979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:06:46.953267 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34003
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:47.302549 4307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:47.303063 4307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:47.303249 4307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:47.311929 4307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:47.312067 4307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:47.312253 4307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:06:47.322641 4307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:47.324038 4307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:47.325562 4307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:47.341727 4312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:47.341671 4313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:47.341708 4315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:47.342657 4307 server_base.cc:1034] running on GCE node
I20260619 01:06:47.343428 4307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:47.344589 4307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:47.345831 4307 hybrid_clock.cc:648] HybridClock initialized: now 1781831207345769 us; error 65 us; skew 500 ppm
I20260619 01:06:47.346267 4307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:47 dist-test-slave-5p8w krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:47.349781 4307 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:06:47.351419 4307 webserver.cc:460] Webserver started at http://127.0.97.3:43851/ using document root <none> and password file <none>
I20260619 01:06:47.352157 4307 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:47.352263 4307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:47.352530 4307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:47.355619 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "48c0c1ce0c904069bc77041ca5601c89"
format_stamp: "Formatted at 2026-06-19 01:06:47 on dist-test-slave-5p8w"
server_key: "acc7b52a5574b3c00d481b189c1ea07d"
server_key_iv: "6a521cdf67d06c4ea024412f47b30f23"
server_key_version: "encryptionkey@0"
I20260619 01:06:47.356412 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "48c0c1ce0c904069bc77041ca5601c89"
format_stamp: "Formatted at 2026-06-19 01:06:47 on dist-test-slave-5p8w"
server_key: "acc7b52a5574b3c00d481b189c1ea07d"
server_key_iv: "6a521cdf67d06c4ea024412f47b30f23"
server_key_version: "encryptionkey@0"
I20260619 01:06:47.362506 4307 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 01:06:47.366561 4322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:47.368219 4307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:06:47.368443 4307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "48c0c1ce0c904069bc77041ca5601c89"
format_stamp: "Formatted at 2026-06-19 01:06:47 on dist-test-slave-5p8w"
server_key: "acc7b52a5574b3c00d481b189c1ea07d"
server_key_iv: "6a521cdf67d06c4ea024412f47b30f23"
server_key_version: "encryptionkey@0"
I20260619 01:06:47.368721 4307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:47.389809 4307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:47.394516 4307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:47.394829 4307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:47.395809 4307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:47.397212 4307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:47.397336 4307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:47.397498 4307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:47.397574 4307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:47.425397 4168 heartbeater.cc:503] Master 127.0.97.62:34003 was elected leader, sending a full tablet report...
I20260619 01:06:47.432955 4307 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38713
I20260619 01:06:47.432992 4434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38713 every 8 connection(s)
I20260619 01:06:47.434440 4307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:06:47.444070 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 4307
I20260619 01:06:47.444278 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:06:47.444736 388 external_mini_cluster.cc:1428] Setting key 86ed9f007f5e99ea27623132b6348a57
Jun 19 01:06:47 dist-test-slave-5p8w krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:47.456465 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:47.437139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50741 (local address 127.0.97.62:34003)
0619 01:06:47.437632 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:47.437636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:47.438734 (+ 1098us) server_negotiation.cc:407] Connection header received
0619 01:06:47.440702 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:47.440708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:47.440802 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:47.440922 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:47.442697 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.443536 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.445561 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.446180 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.446326 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:47.450579 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:47.450585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:47.450588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:47.450630 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:47.453526 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.454298 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.454302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.454305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.454432 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.454859 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.454863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.454865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.455166 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:47.455285 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:47.456002 (+ 717us) server_negotiation.cc:299] Negotiation successful
0619 01:06:47.456177 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":271,"threads_started":1}
I20260619 01:06:47.457541 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:47.437818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34003 (local address 127.0.97.3:50741)
0619 01:06:47.438557 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:47.438605 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:47.440370 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:47.441202 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:47.441211 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:47.441853 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:47.442490 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:47.442509 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.443721 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.443725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:47.445356 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:47.445373 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.446428 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.446432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:47.446769 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:47.447539 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:47.447577 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:47.450286 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:47.453684 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:47.453691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:47.453706 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:47.454154 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:47.454527 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:47.454530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:47.454532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:47.454720 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:47.455291 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:47.455298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:47.455647 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 01:06:47.455883 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 01:06:47.456290 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":306,"threads_started":1}
I20260619 01:06:47.459215 4435 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34003
I20260619 01:06:47.459564 4435 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:47.460327 4435 heartbeater.cc:511] Master 127.0.97.62:34003 requested a full tablet report, sending...
I20260619 01:06:47.461777 3979 ts_manager.cc:194] Registered new tserver with Master: 48c0c1ce0c904069bc77041ca5601c89 (127.0.97.3:38713)
I20260619 01:06:47.462574 3979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50741
I20260619 01:06:47.473215 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 01:06:47 dist-test-slave-5p8w krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 01:06:47 dist-test-slave-5p8w krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:06:47.525204 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:47.509284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.0.97.1:38861)
0619 01:06:47.510017 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:47.510025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:47.510053 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:06:47.510114 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:47.510120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:47.510352 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:47.510507 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:47.511886 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.512968 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.514516 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.515129 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.515238 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:47.519392 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:47.519405 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:47.519420 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:47.519480 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:47.522585 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.523288 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.523296 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.523299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.523453 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.523841 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.523845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.523847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.524219 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:47.524450 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:47.524722 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 01:06:47.524921 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":350,"threads_started":1}
Jun 19 01:06:47 dist-test-slave-5p8w krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:47.544162 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:47.529784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.0.97.62:34003)
0619 01:06:47.530015 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:47.530021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:47.530278 (+ 257us) server_negotiation.cc:407] Connection header received
0619 01:06:47.530493 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:47.530503 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:47.530579 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:47.530696 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:47.531934 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.532685 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.534324 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.534750 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.534894 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:47.539208 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:47.539213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:47.539221 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:47.539262 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:47.541828 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.542512 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.542516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.542519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.542657 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.542973 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.542976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.542978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.543301 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:47.543473 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:47.543766 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 01:06:47.543928 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 01:06:47.544824 3979 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52354
I20260619 01:06:47.546000 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 4040
I20260619 01:06:47.569655 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:47.546984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.0.97.1:38861)
0619 01:06:47.547200 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:47.547207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:47.547384 (+ 177us) server_negotiation.cc:407] Connection header received
0619 01:06:47.547576 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:47.547581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:47.547694 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:47.547803 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:47.549017 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.549786 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.550998 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.551447 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.551599 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:47.563996 (+ 12397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:47.564002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:47.564006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:47.564048 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:47.566539 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.567427 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.567433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.567436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.567586 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.568004 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.568011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.568013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.568411 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:47.568719 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:47.569071 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 01:06:47.569324 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 01:06:47.570283 4163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:47.653141 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 4040
I20260619 01:06:47.670920 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 4174
Jun 19 01:06:47 dist-test-slave-5p8w krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:06:47.687654 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:47.671910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.0.97.2:42617)
0619 01:06:47.672406 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:47.672413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:47.672439 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:06:47.672495 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:47.672500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:47.672733 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:47.672859 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:47.674220 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.675241 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.676451 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.677006 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.677104 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:47.681246 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:47.681259 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:47.681275 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:47.681365 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:47.684345 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.685126 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.685134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.685136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.685266 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.685955 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.685963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.685970 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.686371 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:47.686712 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:47.687008 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 01:06:47.687285 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260619 01:06:47.689052 4297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:47.772522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 4174
I20260619 01:06:47.788861 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 4307
Jun 19 01:06:47 dist-test-slave-5p8w krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:06:47.806015 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:47.789963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.0.97.3:38713)
0619 01:06:47.790601 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:47.790615 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:47.790644 (+ 29us) server_negotiation.cc:407] Connection header received
0619 01:06:47.790705 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:47.790711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:47.790922 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:47.791102 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:47.792435 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.793563 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.794971 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:47.795552 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:47.795709 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:47.800099 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:47.800116 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:47.800137 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:47.800184 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:47.802980 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.803738 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.803745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.803748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.803882 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:47.804277 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:47.804282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:47.804285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:47.804668 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:47.804984 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:47.805377 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 01:06:47.805703 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":332,"threads_started":1}
I20260619 01:06:47.807312 4430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:47.891538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 4307
I20260619 01:06:47.907689 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 3947
I20260619 01:06:47.908934 4009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:48.012686 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 3947
2026-06-19T01:06:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4051 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:06:48 dist-test-slave-5p8w krb5kdc[4455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:06:48 dist-test-slave-5p8w krb5kdc[4455](info): set up 2 sockets
Jun 19 01:06:48 dist-test-slave-5p8w krb5kdc[4455](info): commencing operation
krb5kdc: starting...
W20260619 01:06:49.113755 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:06:49 dist-test-slave-5p8w krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:06:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:06:49.329774 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38287
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:49.664803 4468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:49.665210 4468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:49.665402 4468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:49.673038 4468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:49.673221 4468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:49.673362 4468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:49.673461 4468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:49.673556 4468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:06:49.684134 4468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:49.685624 4468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:49.687052 4468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:49.700268 4473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:49.700353 4476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:49.700372 4474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:49.701326 4468 server_base.cc:1034] running on GCE node
I20260619 01:06:49.702314 4468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:49.703931 4468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:49.705199 4468 hybrid_clock.cc:648] HybridClock initialized: now 1781831209705143 us; error 58 us; skew 500 ppm
I20260619 01:06:49.705739 4468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:49 dist-test-slave-5p8w krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:49.714259 4468 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:06:49.715713 4468 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:49.715883 4468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:49.716284 4468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:49.719693 4468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "5ae3c410ace54e6ba4b3429219b3efe0"
format_stamp: "Formatted at 2026-06-19 01:06:49 on dist-test-slave-5p8w"
server_key: "0ec8360ebf76c984213a86f004ac08e8"
server_key_iv: "106dccd07a8fd5d18677592d5ffe4295"
server_key_version: "encryptionkey@0"
I20260619 01:06:49.720584 4468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "5ae3c410ace54e6ba4b3429219b3efe0"
format_stamp: "Formatted at 2026-06-19 01:06:49 on dist-test-slave-5p8w"
server_key: "0ec8360ebf76c984213a86f004ac08e8"
server_key_iv: "106dccd07a8fd5d18677592d5ffe4295"
server_key_version: "encryptionkey@0"
I20260619 01:06:49.726426 4468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 01:06:49.730408 4482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:49.731969 4468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:06:49.732200 4468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "5ae3c410ace54e6ba4b3429219b3efe0"
format_stamp: "Formatted at 2026-06-19 01:06:49 on dist-test-slave-5p8w"
server_key: "0ec8360ebf76c984213a86f004ac08e8"
server_key_iv: "106dccd07a8fd5d18677592d5ffe4295"
server_key_version: "encryptionkey@0"
I20260619 01:06:49.732461 4468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:49.747139 4468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:49.752627 4468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:49.752949 4468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:49.780921 4468 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38287
I20260619 01:06:49.780882 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38287 every 8 connection(s)
I20260619 01:06:49.782259 4468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:49.786902 4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:49.790225 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 4468
I20260619 01:06:49.790462 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:49.791252 388 external_mini_cluster.cc:1428] Setting key 24e21c24955ce3ae0b10acda2e8622c2
I20260619 01:06:49.804162 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0: Bootstrap starting.
I20260619 01:06:49.808383 4534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:49.809909 4534 log.cc:824] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0: Log is configured to *not* fsync() on all Append() calls
Jun 19 01:06:49 dist-test-slave-5p8w krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:49.813546 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0: No bootstrap required, opened a new log
I20260619 01:06:49.821002 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:49.793754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.0.97.62:38287)
0619 01:06:49.794831 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:49.794853 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:49.794917 (+ 64us) server_negotiation.cc:407] Connection header received
0619 01:06:49.798121 (+ 3204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:49.798166 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:49.798961 (+ 795us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:49.799576 (+ 615us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:49.801637 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:49.803982 (+ 2345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:49.805942 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:49.806715 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:49.806862 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:49.811832 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:49.811843 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:49.811865 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:49.811908 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:49.815836 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:49.816644 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:49.816655 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:49.816666 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:49.816905 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:49.817320 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:49.817325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:49.817329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:49.818151 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:49.818488 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:49.818916 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 01:06:49.819443 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":458,"threads_started":1}
I20260619 01:06:49.823834 4534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae3c410ace54e6ba4b3429219b3efe0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38287 } }
I20260619 01:06:49.824244 4534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:49.824340 4534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae3c410ace54e6ba4b3429219b3efe0, State: Initialized, Role: FOLLOWER
I20260619 01:06:49.825004 4534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae3c410ace54e6ba4b3429219b3efe0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38287 } }
I20260619 01:06:49.825278 4534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:49.825441 4534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:49.825626 4534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:49.827905 4534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae3c410ace54e6ba4b3429219b3efe0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38287 } }
I20260619 01:06:49.828500 4534 leader_election.cc:302] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ae3c410ace54e6ba4b3429219b3efe0; no voters:
I20260619 01:06:49.829175 4534 leader_election.cc:288] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:49.829403 4539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:49.830650 4539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [term 1 LEADER]: Becoming Leader. State: Replica: 5ae3c410ace54e6ba4b3429219b3efe0, State: Running, Role: LEADER
I20260619 01:06:49.831248 4539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae3c410ace54e6ba4b3429219b3efe0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38287 } }
I20260619 01:06:49.831476 4534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:49.834832 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ae3c410ace54e6ba4b3429219b3efe0. Latest consensus state: current_term: 1 leader_uuid: "5ae3c410ace54e6ba4b3429219b3efe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae3c410ace54e6ba4b3429219b3efe0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38287 } } }
I20260619 01:06:49.835180 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:49.834745 4540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ae3c410ace54e6ba4b3429219b3efe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ae3c410ace54e6ba4b3429219b3efe0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38287 } } }
I20260619 01:06:49.835646 4540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:49.836473 4548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:49.842972 4548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:49.854099 4548 catalog_manager.cc:1347] Generated new cluster ID: 237226ad2ab74343a6e00f5eb40e48f7
I20260619 01:06:49.854231 4548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:49.866377 4548 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:49.868423 4548 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:49.879868 4548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0: Generated new TSK 0
I20260619 01:06:49.881121 4548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:06:49.934811 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38287
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:50.287772 4560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:50.288164 4560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:50.288335 4560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:50.296511 4560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:50.296698 4560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:50.296888 4560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:06:50.307366 4560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:50.308858 4560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:50.310392 4560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:50.325884 4568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:50.325918 4566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:50.325832 4565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:50.327095 4560 server_base.cc:1034] running on GCE node
I20260619 01:06:50.327839 4560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:50.328886 4560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:50.330114 4560 hybrid_clock.cc:648] HybridClock initialized: now 1781831210330035 us; error 71 us; skew 500 ppm
I20260619 01:06:50.330479 4560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:50 dist-test-slave-5p8w krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:50.334208 4560 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:06:50.335595 4560 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:50.335757 4560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:50.336181 4560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:50.339649 4560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "4f5ca1d0d66b479083d36b09390b7f61"
format_stamp: "Formatted at 2026-06-19 01:06:50 on dist-test-slave-5p8w"
server_key: "d4abf8e18c5ef10c81d576b60c51c6ae"
server_key_iv: "935e27b75d8fa8866cfaecb7ec380b21"
server_key_version: "encryptionkey@0"
I20260619 01:06:50.340519 4560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4f5ca1d0d66b479083d36b09390b7f61"
format_stamp: "Formatted at 2026-06-19 01:06:50 on dist-test-slave-5p8w"
server_key: "d4abf8e18c5ef10c81d576b60c51c6ae"
server_key_iv: "935e27b75d8fa8866cfaecb7ec380b21"
server_key_version: "encryptionkey@0"
I20260619 01:06:50.346514 4560 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 01:06:50.350574 4574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:50.352151 4560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:06:50.352316 4560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "4f5ca1d0d66b479083d36b09390b7f61"
format_stamp: "Formatted at 2026-06-19 01:06:50 on dist-test-slave-5p8w"
server_key: "d4abf8e18c5ef10c81d576b60c51c6ae"
server_key_iv: "935e27b75d8fa8866cfaecb7ec380b21"
server_key_version: "encryptionkey@0"
I20260619 01:06:50.352533 4560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:50.368691 4560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:50.373626 4560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:50.373867 4560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:50.374850 4560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:50.376204 4560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:50.376289 4560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:50.376371 4560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:50.376430 4560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:50.411067 4560 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44315
I20260619 01:06:50.411105 4686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44315 every 8 connection(s)
I20260619 01:06:50.412039 4560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:06:50.418576 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 4560
I20260619 01:06:50.418792 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:06:50.419411 388 external_mini_cluster.cc:1428] Setting key fe81d2cba674db26abff5c9c267bec84
Jun 19 01:06:50 dist-test-slave-5p8w krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:50.435151 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:50.414782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40677 (local address 127.0.97.62:38287)
0619 01:06:50.415264 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:50.415269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:50.416346 (+ 1077us) server_negotiation.cc:407] Connection header received
0619 01:06:50.418359 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:50.418364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:50.418478 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:50.418609 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:50.420568 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:50.421544 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:50.423277 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:50.424021 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:50.424185 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:50.428316 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:50.428322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:50.428325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:50.428363 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:50.431123 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:50.432313 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:50.432319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:50.432322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:50.432462 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:50.433017 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:50.433021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:50.433023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:50.433370 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:50.433513 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:50.434558 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0619 01:06:50.434769 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":255,"threads_started":1}
I20260619 01:06:50.436949 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:50.415397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38287 (local address 127.0.97.1:40677)
0619 01:06:50.416150 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:50.416200 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:50.418028 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:50.418913 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:50.418928 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:50.419549 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:50.420355 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:50.420377 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:50.421718 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:50.421723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:50.423109 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:50.423123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:50.424318 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:50.424325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:50.424550 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:50.425528 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:50.425559 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:50.428026 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:50.431367 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:50.431381 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:50.431406 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:50.432153 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:50.432590 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:50.432598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:50.432605 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:50.432882 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:50.433517 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:50.433529 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:50.434091 (+ 562us) client_negotiation.cc:769] Sending connection context
0619 01:06:50.434448 (+ 357us) client_negotiation.cc:240] Negotiation successful
0619 01:06:50.435215 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":296,"threads_started":1}
I20260619 01:06:50.438771 4687 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38287
I20260619 01:06:50.439132 4687 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:06:50.439930 4687 heartbeater.cc:511] Master 127.0.97.62:38287 requested a full tablet report, sending...
I20260619 01:06:50.442315 4499 ts_manager.cc:194] Registered new tserver with Master: 4f5ca1d0d66b479083d36b09390b7f61 (127.0.97.1:44315)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:06:50.444638 4499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:06:50.469643 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38287
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:50.819708 4693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:50.820071 4693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:50.820266 4693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:50.828567 4693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:50.828704 4693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:50.828969 4693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:06:50.839926 4693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:50.841388 4693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:50.842798 4693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:50.857189 4698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:50.857172 4699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:50.857242 4701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:50.858139 4693 server_base.cc:1034] running on GCE node
I20260619 01:06:50.858819 4693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:50.859875 4693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:50.861109 4693 hybrid_clock.cc:648] HybridClock initialized: now 1781831210861067 us; error 33 us; skew 500 ppm
I20260619 01:06:50.861524 4693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:50 dist-test-slave-5p8w krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:50.865031 4693 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:06:50.866513 4693 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:50.866632 4693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:50.867035 4693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:50.870313 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "38ba0306c7e64a488c2c3337ba80057d"
format_stamp: "Formatted at 2026-06-19 01:06:50 on dist-test-slave-5p8w"
server_key: "46fe00e3f5c5ff187d779e5408fa4a9d"
server_key_iv: "7fe4c66aa29d8e7bf4a3a0e0b80cd9a4"
server_key_version: "encryptionkey@0"
I20260619 01:06:50.871191 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "38ba0306c7e64a488c2c3337ba80057d"
format_stamp: "Formatted at 2026-06-19 01:06:50 on dist-test-slave-5p8w"
server_key: "46fe00e3f5c5ff187d779e5408fa4a9d"
server_key_iv: "7fe4c66aa29d8e7bf4a3a0e0b80cd9a4"
server_key_version: "encryptionkey@0"
I20260619 01:06:50.876717 4693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 01:06:50.880795 4707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:50.882393 4693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:50.882628 4693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "38ba0306c7e64a488c2c3337ba80057d"
format_stamp: "Formatted at 2026-06-19 01:06:50 on dist-test-slave-5p8w"
server_key: "46fe00e3f5c5ff187d779e5408fa4a9d"
server_key_iv: "7fe4c66aa29d8e7bf4a3a0e0b80cd9a4"
server_key_version: "encryptionkey@0"
I20260619 01:06:50.882946 4693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:50.926481 4693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:50.931732 4693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:50.932086 4693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:50.933197 4693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:50.934778 4693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:50.934900 4693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:50.935051 4693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:50.935148 4693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:50.971235 4693 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37969
I20260619 01:06:50.971256 4819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37969 every 8 connection(s)
I20260619 01:06:50.972154 4693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:06:50.981657 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 4693
I20260619 01:06:50.981839 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:06:50.982287 388 external_mini_cluster.cc:1428] Setting key 6cd42ac9dfefd532575db47e22d060b7
Jun 19 01:06:50 dist-test-slave-5p8w krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:50.996150 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:50.974744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37641 (local address 127.0.97.62:38287)
0619 01:06:50.975207 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:50.975213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:50.976292 (+ 1079us) server_negotiation.cc:407] Connection header received
0619 01:06:50.978227 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:50.978231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:50.978302 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:50.978402 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:50.980110 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:50.981407 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:50.983449 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:50.983993 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:50.984138 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:50.988445 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:50.988451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:50.988454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:50.988490 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:50.992203 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:50.993207 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:50.993215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:50.993221 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:50.993437 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:50.993941 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:50.993949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:50.993956 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:50.994367 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:50.994529 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:50.995315 (+ 786us) server_negotiation.cc:299] Negotiation successful
0619 01:06:50.995626 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
I20260619 01:06:50.996937 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:50.975384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38287 (local address 127.0.97.2:37641)
0619 01:06:50.976114 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:50.976154 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:50.977939 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:50.978647 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:50.978657 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:50.979208 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:50.979921 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:50.979938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:50.981609 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:50.981613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:50.983292 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:50.983308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:50.984250 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:50.984258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:50.984510 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:50.985258 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:50.985283 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:50.988147 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:50.992471 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:50.992488 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:50.992506 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:50.993040 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:50.993586 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:50.993590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:50.993593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:50.993816 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:50.994514 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:50.994522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:50.994912 (+ 390us) client_negotiation.cc:769] Sending connection context
0619 01:06:50.995183 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 01:06:50.995639 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":289,"threads_started":1}
I20260619 01:06:50.998793 4820 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38287
I20260619 01:06:50.999190 4820 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:51.000177 4820 heartbeater.cc:511] Master 127.0.97.62:38287 requested a full tablet report, sending...
I20260619 01:06:51.002211 4499 ts_manager.cc:194] Registered new tserver with Master: 38ba0306c7e64a488c2c3337ba80057d (127.0.97.2:37969)
I20260619 01:06:51.003074 4499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37641
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:06:51.033720 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38287
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:06:51.390021 4826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:51.390349 4826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:51.390538 4826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:51.398077 4826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:51.398212 4826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:51.398435 4826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:06:51.409379 4826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:51.410925 4826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:51.412387 4826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:51.427244 4831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:51.427325 4832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:51.427271 4834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:51.428156 4826 server_base.cc:1034] running on GCE node
I20260619 01:06:51.428804 4826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:51.429957 4826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:51.431237 4826 hybrid_clock.cc:648] HybridClock initialized: now 1781831211431151 us; error 84 us; skew 500 ppm
I20260619 01:06:51.431571 4826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:06:51 dist-test-slave-5p8w krb5kdc[4455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:06:51.435207 4826 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:06:51.436717 4826 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:51.436875 4826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:51.437331 4826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:51.440690 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "b54c74e537224e339678f493707cc6f6"
format_stamp: "Formatted at 2026-06-19 01:06:51 on dist-test-slave-5p8w"
server_key: "6cb2e1aa04aaa94feb57121a1afb0e3e"
server_key_iv: "c4a792575192b61b702e07e40838f695"
server_key_version: "encryptionkey@0"
I20260619 01:06:51.441694 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b54c74e537224e339678f493707cc6f6"
format_stamp: "Formatted at 2026-06-19 01:06:51 on dist-test-slave-5p8w"
server_key: "6cb2e1aa04aaa94feb57121a1afb0e3e"
server_key_iv: "c4a792575192b61b702e07e40838f695"
server_key_version: "encryptionkey@0"
I20260619 01:06:51.448055 4826 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260619 01:06:51.448287 4687 heartbeater.cc:503] Master 127.0.97.62:38287 was elected leader, sending a full tablet report...
I20260619 01:06:51.452667 4840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:51.454277 4826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:51.454452 4826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "b54c74e537224e339678f493707cc6f6"
format_stamp: "Formatted at 2026-06-19 01:06:51 on dist-test-slave-5p8w"
server_key: "6cb2e1aa04aaa94feb57121a1afb0e3e"
server_key_iv: "c4a792575192b61b702e07e40838f695"
server_key_version: "encryptionkey@0"
I20260619 01:06:51.454744 4826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:51.476826 4826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:51.481485 4826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:51.481801 4826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:51.482864 4826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:51.484416 4826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:51.484509 4826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:51.484688 4826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:51.484786 4826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:51.520165 4826 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40675
I20260619 01:06:51.520205 4952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40675 every 8 connection(s)
I20260619 01:06:51.521106 4826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:06:51.526088 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 4826
I20260619 01:06:51.526293 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:06:51.526916 388 external_mini_cluster.cc:1428] Setting key 4698cb802e808365c17d383030d12414
Jun 19 01:06:51 dist-test-slave-5p8w krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:06:51.545046 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.523977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49977 (local address 127.0.97.62:38287)
0619 01:06:51.524442 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:51.524447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:51.525840 (+ 1393us) server_negotiation.cc:407] Connection header received
0619 01:06:51.528806 (+ 2966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:51.528810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:51.528882 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:51.528986 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:51.531070 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.531984 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.533870 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.534303 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.534409 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.538404 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:51.538410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:51.538413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:51.538453 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:51.541278 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:51.542441 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:51.542446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:51.542449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:51.542629 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:51.543336 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:51.543341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:51.543350 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:51.543647 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:51.543776 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:51.544582 (+ 806us) server_negotiation.cc:299] Negotiation successful
0619 01:06:51.544774 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":233,"threads_started":1}
I20260619 01:06:51.546160 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.524633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38287 (local address 127.0.97.3:49977)
0619 01:06:51.525587 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:51.525669 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:51.528395 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:51.529297 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:51.529341 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:51.530000 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:51.530882 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.530904 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.532156 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.532160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.533696 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.533711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.534564 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.534575 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.534802 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.535490 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:06:51.535516 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:51.538093 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:51.541492 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:51.541503 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:51.541527 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:51.542284 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:51.542745 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:06:51.542749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:06:51.542751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:06:51.543058 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:06:51.543786 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:51.543793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:51.544172 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 01:06:51.544431 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 01:06:51.544873 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":339,"threads_started":1}
I20260619 01:06:51.547900 4953 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38287
I20260619 01:06:51.548249 4953 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:51.548970 4953 heartbeater.cc:511] Master 127.0.97.62:38287 requested a full tablet report, sending...
I20260619 01:06:51.550631 4499 ts_manager.cc:194] Registered new tserver with Master: b54c74e537224e339678f493707cc6f6 (127.0.97.3:40675)
I20260619 01:06:51.551460 4499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49977
I20260619 01:06:51.555856 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:06:51.573720 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.560865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.0.97.62:38287)
0619 01:06:51.561086 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:51.561091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:51.561661 (+ 570us) server_negotiation.cc:407] Connection header received
0619 01:06:51.561826 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:51.561830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:51.561899 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:51.562010 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:51.563498 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.564229 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.565609 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.566049 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.566156 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.568553 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:51.568558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:51.568562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:51.568600 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:51.571246 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:51.571966 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:51.571987 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:51.571989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:51.572108 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:51.572431 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:51.572434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:51.572436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:51.572766 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:51.572943 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:51.573237 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 01:06:51.573479 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 01:06:51.578397 4499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:06:51.585789 4499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:06:51.623279 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.599619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44315 (local address 127.0.0.1:57446)
0619 01:06:51.613364 (+ 13745us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:51.613407 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:51.613623 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:51.615116 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:51.615124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:51.615151 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:51.615613 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.615628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.617701 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.617709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.619924 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.619940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.622153 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.622161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.622374 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.622437 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 01:06:51.622578 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 01:06:51.622853 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13447,"mutex_wait_us":1,"thread_start_us":15896,"threads_started":1}
I20260619 01:06:51.623744 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.599983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.0.97.1:44315)
0619 01:06:51.614331 (+ 14348us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:51.614341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:51.614378 (+ 37us) server_negotiation.cc:407] Connection header received
0619 01:06:51.614520 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:51.614533 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:51.614858 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:51.615851 (+ 993us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:51.616103 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.617564 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.620124 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.621830 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.621974 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.622251 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:51.622914 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 01:06:51.623107 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14131,"thread_start_us":14073,"threads_started":1}
I20260619 01:06:51.624789 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.599630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40675 (local address 127.0.0.1:41726)
0619 01:06:51.614869 (+ 15239us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:51.614892 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:51.615124 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:51.616110 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:51.616114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:51.616139 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:51.616650 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.616663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.618723 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.618728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.620897 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.620919 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.622655 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.622661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.622878 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.624085 (+ 1207us) client_negotiation.cc:769] Sending connection context
0619 01:06:51.624183 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 01:06:51.624353 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15098,"mutex_wait_us":732,"thread_start_us":12603,"threads_started":1}
I20260619 01:06:51.624878 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.599950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.0.97.3:40675)
0619 01:06:51.612390 (+ 12440us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:51.612407 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:51.615018 (+ 2611us) server_negotiation.cc:407] Connection header received
0619 01:06:51.615334 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:51.615342 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:51.615599 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:51.615780 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:51.616882 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.618546 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.621079 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.622436 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.622578 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.622791 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:51.624349 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0619 01:06:51.624545 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12147,"thread_start_us":12490,"threads_started":1}
I20260619 01:06:51.625365 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.602272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37969 (local address 127.0.0.1:49572)
0619 01:06:51.614954 (+ 12682us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:51.614972 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:51.615186 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:51.616243 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:51.616247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:51.616265 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:51.616653 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.616660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.618723 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.618728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.622100 (+ 3372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.622115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.624426 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.624433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.624615 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.624654 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 01:06:51.624761 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 01:06:51.624929 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12544,"mutex_wait_us":119,"thread_start_us":12261,"threads_started":1}
I20260619 01:06:51.626382 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.602589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.0.97.2:37969)
0619 01:06:51.614141 (+ 11552us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:51.614151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:51.615085 (+ 934us) server_negotiation.cc:407] Connection header received
0619 01:06:51.615374 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:51.615382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:51.615903 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:51.616033 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:51.616816 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.618537 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.622307 (+ 3770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.623717 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.623799 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.624057 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:51.625849 (+ 1792us) server_negotiation.cc:299] Negotiation successful
0619 01:06:51.626050 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11367,"thread_start_us":11213,"threads_started":1}
I20260619 01:06:51.628345 4888 tablet_service.cc:1467] Processing CreateTablet for tablet fafdf93fad324c6b9f21bbb59bc4575f (DEFAULT_TABLE table=test-table [id=55d862c966b64dc2bec51fb1c2103e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:51.628373 4622 tablet_service.cc:1467] Processing CreateTablet for tablet fafdf93fad324c6b9f21bbb59bc4575f (DEFAULT_TABLE table=test-table [id=55d862c966b64dc2bec51fb1c2103e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:51.629945 4888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafdf93fad324c6b9f21bbb59bc4575f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:51.629964 4622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafdf93fad324c6b9f21bbb59bc4575f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:51.630378 4755 tablet_service.cc:1467] Processing CreateTablet for tablet fafdf93fad324c6b9f21bbb59bc4575f (DEFAULT_TABLE table=test-table [id=55d862c966b64dc2bec51fb1c2103e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:51.631912 4755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafdf93fad324c6b9f21bbb59bc4575f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:51.644593 4972 tablet_bootstrap.cc:492] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: Bootstrap starting.
I20260619 01:06:51.646795 4973 tablet_bootstrap.cc:492] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: Bootstrap starting.
I20260619 01:06:51.648720 4972 tablet_bootstrap.cc:654] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:51.648729 4974 tablet_bootstrap.cc:492] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: Bootstrap starting.
I20260619 01:06:51.650604 4973 tablet_bootstrap.cc:654] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:51.650681 4972 log.cc:824] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:51.651872 4974 tablet_bootstrap.cc:654] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:51.652333 4973 log.cc:824] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:51.653155 4974 log.cc:824] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:51.655170 4972 tablet_bootstrap.cc:492] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: No bootstrap required, opened a new log
I20260619 01:06:51.655493 4972 ts_tablet_manager.cc:1397] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260619 01:06:51.656199 4973 tablet_bootstrap.cc:492] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: No bootstrap required, opened a new log
I20260619 01:06:51.656301 4974 tablet_bootstrap.cc:492] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: No bootstrap required, opened a new log
I20260619 01:06:51.656491 4973 ts_tablet_manager.cc:1397] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 01:06:51.656579 4974 ts_tablet_manager.cc:1397] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 01:06:51.665242 4973 raft_consensus.cc:348] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54c74e537224e339678f493707cc6f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40675 } } peers { permanent_uuid: "4f5ca1d0d66b479083d36b09390b7f61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44315 } } peers { permanent_uuid: "38ba0306c7e64a488c2c3337ba80057d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37969 } }
I20260619 01:06:51.665771 4973 raft_consensus.cc:374] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:51.665877 4973 raft_consensus.cc:729] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5ca1d0d66b479083d36b09390b7f61, State: Initialized, Role: FOLLOWER
I20260619 01:06:51.666568 4973 consensus_queue.cc:260] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54c74e537224e339678f493707cc6f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40675 } } peers { permanent_uuid: "4f5ca1d0d66b479083d36b09390b7f61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44315 } } peers { permanent_uuid: "38ba0306c7e64a488c2c3337ba80057d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37969 } }
I20260619 01:06:51.667522 4972 raft_consensus.cc:348] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54c74e537224e339678f493707cc6f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40675 } } peers { permanent_uuid: "4f5ca1d0d66b479083d36b09390b7f61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44315 } } peers { permanent_uuid: "38ba0306c7e64a488c2c3337ba80057d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37969 } }
I20260619 01:06:51.667516 4974 raft_consensus.cc:348] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54c74e537224e339678f493707cc6f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40675 } } peers { permanent_uuid: "4f5ca1d0d66b479083d36b09390b7f61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44315 } } peers { permanent_uuid: "38ba0306c7e64a488c2c3337ba80057d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37969 } }
I20260619 01:06:51.667959 4972 raft_consensus.cc:374] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:51.667958 4974 raft_consensus.cc:374] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:51.668056 4974 raft_consensus.cc:729] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ba0306c7e64a488c2c3337ba80057d, State: Initialized, Role: FOLLOWER
I20260619 01:06:51.668056 4972 raft_consensus.cc:729] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54c74e537224e339678f493707cc6f6, State: Initialized, Role: FOLLOWER
I20260619 01:06:51.668507 4973 ts_tablet_manager.cc:1428] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 01:06:51.668708 4974 consensus_queue.cc:260] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54c74e537224e339678f493707cc6f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40675 } } peers { permanent_uuid: "4f5ca1d0d66b479083d36b09390b7f61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44315 } } peers { permanent_uuid: "38ba0306c7e64a488c2c3337ba80057d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37969 } }
I20260619 01:06:51.668881 4972 consensus_queue.cc:260] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54c74e537224e339678f493707cc6f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40675 } } peers { permanent_uuid: "4f5ca1d0d66b479083d36b09390b7f61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44315 } } peers { permanent_uuid: "38ba0306c7e64a488c2c3337ba80057d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37969 } }
I20260619 01:06:51.670017 4820 heartbeater.cc:503] Master 127.0.97.62:38287 was elected leader, sending a full tablet report...
I20260619 01:06:51.670775 4974 ts_tablet_manager.cc:1428] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260619 01:06:51.672925 4953 heartbeater.cc:503] Master 127.0.97.62:38287 was elected leader, sending a full tablet report...
I20260619 01:06:51.673683 4972 ts_tablet_manager.cc:1428] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: Time spent starting tablet: real 0.018s user 0.014s sys 0.002s
W20260619 01:06:51.724285 4821 tablet.cc:2369] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:06:51.772950 4954 tablet.cc:2369] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:06:51.857164 4978 raft_consensus.cc:482] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:51.857568 4978 raft_consensus.cc:504] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54c74e537224e339678f493707cc6f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40675 } } peers { permanent_uuid: "4f5ca1d0d66b479083d36b09390b7f61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44315 } } peers { permanent_uuid: "38ba0306c7e64a488c2c3337ba80057d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37969 } }
I20260619 01:06:51.859628 4978 leader_election.cc:288] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b54c74e537224e339678f493707cc6f6 (127.0.97.3:40675), 38ba0306c7e64a488c2c3337ba80057d (127.0.97.2:37969)
I20260619 01:06:51.869148 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.859872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34797 (local address 127.0.97.3:40675)
0619 01:06:51.860144 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:51.860154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:51.860931 (+ 777us) server_negotiation.cc:407] Connection header received
0619 01:06:51.861186 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:51.861194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:51.861281 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:51.861405 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:51.862172 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.863323 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.866522 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.867844 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.868006 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.868116 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:51.868711 (+ 595us) server_negotiation.cc:299] Negotiation successful
0619 01:06:51.868866 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 01:06:51.869148 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.860235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38363 (local address 127.0.97.2:37969)
0619 01:06:51.860485 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:51.860506 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:51.861168 (+ 662us) server_negotiation.cc:407] Connection header received
0619 01:06:51.861265 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:51.861270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:51.861387 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:51.861474 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:06:51.862172 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.863324 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.866522 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.867833 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.868012 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.868116 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:51.868711 (+ 595us) server_negotiation.cc:299] Negotiation successful
0619 01:06:51.868892 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 01:06:51.869408 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.859907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37969 (local address 127.0.97.1:38363)
0619 01:06:51.861049 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:51.861066 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:51.861172 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:51.861496 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:51.861502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:51.861526 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:51.862020 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.862033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.863501 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.863505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.866281 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.866303 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.868170 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.868175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.868409 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.868497 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 01:06:51.868633 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 01:06:51.868874 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"mutex_wait_us":305,"thread_start_us":782,"threads_started":1}
I20260619 01:06:51.869477 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.859823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40675 (local address 127.0.97.1:34797)
0619 01:06:51.860755 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:51.860784 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:51.860986 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:51.861418 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:51.861421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:51.861474 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:06:51.861970 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.861984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.863500 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.863506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.866285 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:51.866303 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.868159 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.868167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:51.868408 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.868490 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 01:06:51.868633 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 01:06:51.868874 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":236,"thread_start_us":528,"threads_started":1}
I20260619 01:06:51.870541 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafdf93fad324c6b9f21bbb59bc4575f" candidate_uuid: "4f5ca1d0d66b479083d36b09390b7f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54c74e537224e339678f493707cc6f6" is_pre_election: true
I20260619 01:06:51.871066 4908 raft_consensus.cc:2454] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5ca1d0d66b479083d36b09390b7f61 in term 0.
I20260619 01:06:51.871816 4575 leader_election.cc:302] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5ca1d0d66b479083d36b09390b7f61, b54c74e537224e339678f493707cc6f6; no voters:
I20260619 01:06:51.871846 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafdf93fad324c6b9f21bbb59bc4575f" candidate_uuid: "4f5ca1d0d66b479083d36b09390b7f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ba0306c7e64a488c2c3337ba80057d" is_pre_election: true
I20260619 01:06:51.872270 4978 raft_consensus.cc:2781] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:06:51.872414 4978 raft_consensus.cc:482] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:06:51.872368 4775 raft_consensus.cc:2454] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5ca1d0d66b479083d36b09390b7f61 in term 0.
I20260619 01:06:51.872507 4978 raft_consensus.cc:3037] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:51.875232 4978 raft_consensus.cc:504] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54c74e537224e339678f493707cc6f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40675 } } peers { permanent_uuid: "4f5ca1d0d66b479083d36b09390b7f61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44315 } } peers { permanent_uuid: "38ba0306c7e64a488c2c3337ba80057d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37969 } }
I20260619 01:06:51.875926 4978 leader_election.cc:288] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [CANDIDATE]: Term 1 election: Requested vote from peers b54c74e537224e339678f493707cc6f6 (127.0.97.3:40675), 38ba0306c7e64a488c2c3337ba80057d (127.0.97.2:37969)
I20260619 01:06:51.876447 4908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafdf93fad324c6b9f21bbb59bc4575f" candidate_uuid: "4f5ca1d0d66b479083d36b09390b7f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54c74e537224e339678f493707cc6f6"
I20260619 01:06:51.876525 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fafdf93fad324c6b9f21bbb59bc4575f" candidate_uuid: "4f5ca1d0d66b479083d36b09390b7f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ba0306c7e64a488c2c3337ba80057d"
I20260619 01:06:51.876660 4908 raft_consensus.cc:3037] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:51.876814 4775 raft_consensus.cc:3037] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:51.879066 4908 raft_consensus.cc:2454] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5ca1d0d66b479083d36b09390b7f61 in term 1.
I20260619 01:06:51.879081 4775 raft_consensus.cc:2454] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5ca1d0d66b479083d36b09390b7f61 in term 1.
I20260619 01:06:51.879617 4576 leader_election.cc:302] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ba0306c7e64a488c2c3337ba80057d, 4f5ca1d0d66b479083d36b09390b7f61; no voters:
I20260619 01:06:51.880023 4978 raft_consensus.cc:2781] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:51.880501 4978 raft_consensus.cc:686] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 1 LEADER]: Becoming Leader. State: Replica: 4f5ca1d0d66b479083d36b09390b7f61, State: Running, Role: LEADER
I20260619 01:06:51.881043 4978 consensus_queue.cc:237] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54c74e537224e339678f493707cc6f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40675 } } peers { permanent_uuid: "4f5ca1d0d66b479083d36b09390b7f61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44315 } } peers { permanent_uuid: "38ba0306c7e64a488c2c3337ba80057d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37969 } }
I20260619 01:06:51.886761 4497 catalog_manager.cc:5495] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5ca1d0d66b479083d36b09390b7f61 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4f5ca1d0d66b479083d36b09390b7f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b54c74e537224e339678f493707cc6f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5ca1d0d66b479083d36b09390b7f61" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38ba0306c7e64a488c2c3337ba80057d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37969 } health_report { overall_health: UNKNOWN } } }
W20260619 01:06:51.914686 4688 tablet.cc:2369] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:06:51.922417 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.916524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.0.97.1:44315)
0619 01:06:51.916774 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:51.916778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:51.916797 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:06:51.916977 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:51.916981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:51.917069 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:51.917197 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:06:51.917816 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.918548 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.920047 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:51.920451 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:51.920554 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:51.921114 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:06:51.921757 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:06:51.921871 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:51.922039 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 01:06:51.922112 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 01:06:51.931504 4775 raft_consensus.cc:1261] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d [term 1 FOLLOWER]: Refusing update from remote peer 4f5ca1d0d66b479083d36b09390b7f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:51.931543 4908 raft_consensus.cc:1261] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6 [term 1 FOLLOWER]: Refusing update from remote peer 4f5ca1d0d66b479083d36b09390b7f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:51.932689 4983 consensus_queue.cc:1035] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b54c74e537224e339678f493707cc6f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:51.933174 4978 consensus_queue.cc:1035] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ba0306c7e64a488c2c3337ba80057d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:51.970625 4499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:51.970988 4499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:51.975529 4499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ae3c410ace54e6ba4b3429219b3efe0: Sending DeleteTablet for 3 replicas of tablet fafdf93fad324c6b9f21bbb59bc4575f
I20260619 01:06:51.976881 4622 tablet_service.cc:1514] Processing DeleteTablet for tablet fafdf93fad324c6b9f21bbb59bc4575f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57446
I20260619 01:06:51.977067 4755 tablet_service.cc:1514] Processing DeleteTablet for tablet fafdf93fad324c6b9f21bbb59bc4575f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49572
I20260619 01:06:51.977742 4888 tablet_service.cc:1514] Processing DeleteTablet for tablet fafdf93fad324c6b9f21bbb59bc4575f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41726
I20260619 01:06:51.979189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 4560
I20260619 01:06:51.999809 4998 tablet_replica.cc:330] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: stopping tablet replica
I20260619 01:06:52.000430 4998 raft_consensus.cc:2229] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:52.001142 4998 raft_consensus.cc:2258] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:52.001283 4996 tablet_replica.cc:330] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: stopping tablet replica
I20260619 01:06:52.001791 4996 raft_consensus.cc:2229] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:06:52.002535 4996 raft_consensus.cc:2258] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:52.003860 4998 ts_tablet_manager.cc:1905] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:52.004793 4996 ts_tablet_manager.cc:1905] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:52.005796 4997 tablet_replica.cc:330] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: stopping tablet replica
I20260619 01:06:52.006253 4997 raft_consensus.cc:2229] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:52.006793 4997 raft_consensus.cc:2258] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 01:06:52 dist-test-slave-5p8w krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:06:52.012157 4998 ts_tablet_manager.cc:1918] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:06:52.012475 4998 log.cc:1196] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/fafdf93fad324c6b9f21bbb59bc4575f
I20260619 01:06:52.012763 4996 ts_tablet_manager.cc:1918] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:06:52.012960 4996 log.cc:1196] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/fafdf93fad324c6b9f21bbb59bc4575f
I20260619 01:06:52.013264 4998 ts_tablet_manager.cc:1939] T fafdf93fad324c6b9f21bbb59bc4575f P b54c74e537224e339678f493707cc6f6: Deleting consensus metadata
I20260619 01:06:52.013654 4996 ts_tablet_manager.cc:1939] T fafdf93fad324c6b9f21bbb59bc4575f P 4f5ca1d0d66b479083d36b09390b7f61: Deleting consensus metadata
I20260619 01:06:52.015506 4483 catalog_manager.cc:4841] TS b54c74e537224e339678f493707cc6f6 (127.0.97.3:40675): tablet fafdf93fad324c6b9f21bbb59bc4575f (table test-table [id=55d862c966b64dc2bec51fb1c2103e0a]) successfully deleted
I20260619 01:06:52.016155 4485 catalog_manager.cc:4841] TS 4f5ca1d0d66b479083d36b09390b7f61 (127.0.97.1:44315): tablet fafdf93fad324c6b9f21bbb59bc4575f (table test-table [id=55d862c966b64dc2bec51fb1c2103e0a]) successfully deleted
I20260619 01:06:52.017582 4997 ts_tablet_manager.cc:1905] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:52.019682 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:51.981845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.0.97.1:44315)
0619 01:06:51.982448 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:51.982461 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:51.999197 (+ 16736us) server_negotiation.cc:407] Connection header received
0619 01:06:51.999400 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:51.999408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:51.999536 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:51.999667 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:52.001903 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:52.003169 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:52.005128 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:52.005869 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:52.005987 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:52.011775 (+ 5788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:52.011808 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:52.011833 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:52.011887 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:52.016356 (+ 4469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:52.017158 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:52.017173 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:52.017176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:52.017424 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:52.017874 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:52.017879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:52.017883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:52.018254 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:52.018564 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:52.018900 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 01:06:52.019174 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434}
I20260619 01:06:52.020383 4682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:52.025342 4997 ts_tablet_manager.cc:1918] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:06:52.025592 4997 log.cc:1196] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/fafdf93fad324c6b9f21bbb59bc4575f
I20260619 01:06:52.026084 4997 ts_tablet_manager.cc:1939] T fafdf93fad324c6b9f21bbb59bc4575f P 38ba0306c7e64a488c2c3337ba80057d: Deleting consensus metadata
I20260619 01:06:52.027812 4484 catalog_manager.cc:4841] TS 38ba0306c7e64a488c2c3337ba80057d (127.0.97.2:37969): tablet fafdf93fad324c6b9f21bbb59bc4575f (table test-table [id=55d862c966b64dc2bec51fb1c2103e0a]) successfully deleted
I20260619 01:06:52.123242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 4560
I20260619 01:06:52.140102 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 4693
Jun 19 01:06:52 dist-test-slave-5p8w krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:06:52.154882 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:52.140996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.0.97.2:37969)
0619 01:06:52.141182 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:52.141186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:52.141207 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:06:52.141265 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:52.141269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:52.141396 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:52.141489 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:52.142699 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:52.143460 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:52.144749 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:52.145146 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:52.145244 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:52.149440 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:52.149459 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:52.149478 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:52.149527 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:52.152214 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:52.153024 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:52.153031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:52.153034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:52.153161 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:52.153548 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:52.153552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:52.153554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:52.153913 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:52.154161 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:52.154463 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 01:06:52.154628 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:06:52.155702 4815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:52.251075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 4693
I20260619 01:06:52.268685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 4826
Jun 19 01:06:52 dist-test-slave-5p8w krb5kdc[4455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:06:52.285117 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:52.269484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.0.97.3:40675)
0619 01:06:52.269669 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:52.269676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:52.269750 (+ 74us) server_negotiation.cc:407] Connection header received
0619 01:06:52.269998 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:52.270002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:52.270071 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:52.270185 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:52.271490 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:52.272226 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:52.273661 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:52.274122 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:52.274264 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:52.279061 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:52.279079 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:52.279096 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:06:52.279147 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:52.282082 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:52.282948 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:52.282957 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:52.282959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:52.283103 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:06:52.283520 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:06:52.283525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:06:52.283527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:06:52.283898 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:52.284203 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:52.284528 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 01:06:52.284746 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:06:52.285854 4948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:52.379319 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 4826
I20260619 01:06:52.396548 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 4468
I20260619 01:06:52.397876 4529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:52.508016 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 4468
2026-06-19T01:06:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4496 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T01:06:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:52Z Disabled control of system clock
I20260619 01:06:52.579422 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42111
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 01:06:52.927139 5007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:52.927461 5007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:52.927616 5007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:52.935261 5007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:52.935392 5007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:52.935494 5007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:52.935585 5007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:52.935654 5007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:06:52.945421 5007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:52.946864 5007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:52.948833 5007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:52.961833 5013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:52.961858 5015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:52.962239 5012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:52.962692 5007 server_base.cc:1034] running on GCE node
I20260619 01:06:52.963560 5007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:52.965224 5007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:52.966498 5007 hybrid_clock.cc:648] HybridClock initialized: now 1781831212966449 us; error 53 us; skew 500 ppm
I20260619 01:06:52.966878 5007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:06:52.974884 5007 webserver.cc:460] Webserver started at http://127.0.97.62:40895/ using document root <none> and password file <none>
I20260619 01:06:52.975786 5007 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:52.975939 5007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:52.976315 5007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:52.979560 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "2dd127638c9d456182d26d708d1f3c30"
format_stamp: "Formatted at 2026-06-19 01:06:52 on dist-test-slave-5p8w"
server_key: "2f145ed6db51908388c43e425a7d7484"
server_key_iv: "23d32964cf264d8631717bc2d4310a2e"
server_key_version: "encryptionkey@0"
I20260619 01:06:52.980453 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "2dd127638c9d456182d26d708d1f3c30"
format_stamp: "Formatted at 2026-06-19 01:06:52 on dist-test-slave-5p8w"
server_key: "2f145ed6db51908388c43e425a7d7484"
server_key_iv: "23d32964cf264d8631717bc2d4310a2e"
server_key_version: "encryptionkey@0"
I20260619 01:06:52.986244 5007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 01:06:52.990155 5021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:52.991710 5007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:06:52.991958 5007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "2dd127638c9d456182d26d708d1f3c30"
format_stamp: "Formatted at 2026-06-19 01:06:52 on dist-test-slave-5p8w"
server_key: "2f145ed6db51908388c43e425a7d7484"
server_key_iv: "23d32964cf264d8631717bc2d4310a2e"
server_key_version: "encryptionkey@0"
I20260619 01:06:52.992226 5007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:52.998790 5007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:52.999732 5007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:53.000070 5007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:53.019829 5007 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42111
I20260619 01:06:53.019932 5072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42111 every 8 connection(s)
I20260619 01:06:53.021411 5007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:53.025646 5073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:53.030524 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 5007
I20260619 01:06:53.030769 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:53.031543 388 external_mini_cluster.cc:1428] Setting key 053e74fcf17bbaa9a2ee146870575eae
I20260619 01:06:53.038841 5073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30: Bootstrap starting.
I20260619 01:06:53.041488 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:53.033940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.0.97.62:42111)
0619 01:06:53.035109 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:53.035122 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:53.035171 (+ 49us) server_negotiation.cc:407] Connection header received
0619 01:06:53.037087 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:53.037117 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:53.037768 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:53.038258 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:53.039509 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:53.039517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:53.039532 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:53.039549 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:53.039619 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:53.039637 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:53.039899 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:53.040064 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 01:06:53.040365 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":383,"threads_started":1}
I20260619 01:06:53.042934 5073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:53.044199 5073 log.cc:824] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:53.047267 5073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30: No bootstrap required, opened a new log
I20260619 01:06:53.055220 5073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd127638c9d456182d26d708d1f3c30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42111 } }
I20260619 01:06:53.055650 5073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:53.055750 5073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd127638c9d456182d26d708d1f3c30, State: Initialized, Role: FOLLOWER
I20260619 01:06:53.056447 5073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd127638c9d456182d26d708d1f3c30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42111 } }
I20260619 01:06:53.056689 5073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:53.056823 5073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:53.057009 5073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:53.059320 5073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd127638c9d456182d26d708d1f3c30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42111 } }
I20260619 01:06:53.059902 5073 leader_election.cc:302] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dd127638c9d456182d26d708d1f3c30; no voters:
I20260619 01:06:53.060490 5073 leader_election.cc:288] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:53.060654 5078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:53.061834 5078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [term 1 LEADER]: Becoming Leader. State: Replica: 2dd127638c9d456182d26d708d1f3c30, State: Running, Role: LEADER
I20260619 01:06:53.062527 5078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd127638c9d456182d26d708d1f3c30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42111 } }
I20260619 01:06:53.063071 5073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:53.065523 5080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd127638c9d456182d26d708d1f3c30. Latest consensus state: current_term: 1 leader_uuid: "2dd127638c9d456182d26d708d1f3c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd127638c9d456182d26d708d1f3c30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42111 } } }
I20260619 01:06:53.066017 5080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:53.065584 5079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dd127638c9d456182d26d708d1f3c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd127638c9d456182d26d708d1f3c30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42111 } } }
I20260619 01:06:53.066349 5079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:53.068198 5087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:53.074046 5087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:53.084352 5087 catalog_manager.cc:1347] Generated new cluster ID: 2de05348a1784d1282cc0ebfe35fcfdb
I20260619 01:06:53.084553 5087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:53.109494 5087 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:53.110014 5087 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:53.118034 5087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30: Generated new TSK 0
I20260619 01:06:53.119206 5087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:06:53.135314 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42111
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 01:06:53.484205 5097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:53.484535 5097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:53.484628 5097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:53.492332 5097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:53.492465 5097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:53.492579 5097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:06:53.502270 5097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:53.503643 5097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:53.505534 5097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:53.519724 5102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:53.519794 5103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:53.519804 5105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:53.520599 5097 server_base.cc:1034] running on GCE node
I20260619 01:06:53.521355 5097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:53.522464 5097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:53.523702 5097 hybrid_clock.cc:648] HybridClock initialized: now 1781831213523645 us; error 85 us; skew 500 ppm
I20260619 01:06:53.524053 5097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:06:53.526544 5097 webserver.cc:460] Webserver started at http://127.0.97.1:42147/ using document root <none> and password file <none>
I20260619 01:06:53.527316 5097 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:53.527465 5097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:53.527803 5097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:53.531049 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "7fb545544c8a4785aa96c87654292b27"
format_stamp: "Formatted at 2026-06-19 01:06:53 on dist-test-slave-5p8w"
server_key: "a187a78ed4733dcbed92947bea316c20"
server_key_iv: "0e0bed5345a4abe5707e26d29d1db7e9"
server_key_version: "encryptionkey@0"
I20260619 01:06:53.531939 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7fb545544c8a4785aa96c87654292b27"
format_stamp: "Formatted at 2026-06-19 01:06:53 on dist-test-slave-5p8w"
server_key: "a187a78ed4733dcbed92947bea316c20"
server_key_iv: "0e0bed5345a4abe5707e26d29d1db7e9"
server_key_version: "encryptionkey@0"
I20260619 01:06:53.537593 5097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 01:06:53.541530 5111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:53.543159 5097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:06:53.543375 5097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "7fb545544c8a4785aa96c87654292b27"
format_stamp: "Formatted at 2026-06-19 01:06:53 on dist-test-slave-5p8w"
server_key: "a187a78ed4733dcbed92947bea316c20"
server_key_iv: "0e0bed5345a4abe5707e26d29d1db7e9"
server_key_version: "encryptionkey@0"
I20260619 01:06:53.543658 5097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:53.550482 5097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:53.551456 5097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:53.551753 5097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:53.552825 5097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:53.554479 5097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:53.554610 5097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:53.554741 5097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:53.554831 5097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:53.588912 5097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43287
I20260619 01:06:53.588958 5223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43287 every 8 connection(s)
I20260619 01:06:53.590445 5097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:06:53.597846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 5097
I20260619 01:06:53.598119 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:06:53.598941 388 external_mini_cluster.cc:1428] Setting key 8bad8da4fe5917e1c7b8be51c01b460a
I20260619 01:06:53.599398 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:53.593141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37063 (local address 127.0.97.62:42111)
0619 01:06:53.593643 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:53.593649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:53.594905 (+ 1256us) server_negotiation.cc:407] Connection header received
0619 01:06:53.596714 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:53.596718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:53.596819 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:53.596930 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:53.598620 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:53.598624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:53.598628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:53.598645 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:53.598687 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:53.598699 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:53.598804 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:53.599122 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 01:06:53.599177 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
I20260619 01:06:53.600765 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:53.593798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42111 (local address 127.0.97.1:37063)
0619 01:06:53.594726 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:53.594770 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:53.596391 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:53.597169 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:53.597177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:53.597339 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:53.598211 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:06:53.598214 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:53.598280 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:06:53.598420 (+ 140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:53.598805 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:53.598817 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:53.598827 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 01:06:53.599157 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 01:06:53.599457 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":276,"threads_started":1}
I20260619 01:06:53.602136 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42111
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 01:06:53.602404 5224 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42111
I20260619 01:06:53.602664 5224 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:53.603446 5224 heartbeater.cc:511] Master 127.0.97.62:42111 requested a full tablet report, sending...
I20260619 01:06:53.606093 5038 ts_manager.cc:194] Registered new tserver with Master: 7fb545544c8a4785aa96c87654292b27 (127.0.97.1:43287)
W20260619 01:06:53.950733 5228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:53.951069 5228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:53.951212 5228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:53.959021 5228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:53.959144 5228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:53.959322 5228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:06:53.969461 5228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:53.970858 5228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:53.973106 5228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:53.988580 5233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:53.988551 5234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:53.988646 5236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:53.989609 5228 server_base.cc:1034] running on GCE node
I20260619 01:06:53.990334 5228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:53.991427 5228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:53.992683 5228 hybrid_clock.cc:648] HybridClock initialized: now 1781831213992625 us; error 62 us; skew 500 ppm
I20260619 01:06:53.993052 5228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:06:53.995689 5228 webserver.cc:460] Webserver started at http://127.0.97.2:45165/ using document root <none> and password file <none>
I20260619 01:06:53.996515 5228 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:53.996666 5228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:53.997087 5228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:54.000836 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "5743f431137e4198b84e005bedf110a1"
format_stamp: "Formatted at 2026-06-19 01:06:53 on dist-test-slave-5p8w"
server_key: "9ce6b0ef2d83a0364de953057f1d3e77"
server_key_iv: "e771000c795d7ebdc634d97765fbcb71"
server_key_version: "encryptionkey@0"
I20260619 01:06:54.001900 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5743f431137e4198b84e005bedf110a1"
format_stamp: "Formatted at 2026-06-19 01:06:53 on dist-test-slave-5p8w"
server_key: "9ce6b0ef2d83a0364de953057f1d3e77"
server_key_iv: "e771000c795d7ebdc634d97765fbcb71"
server_key_version: "encryptionkey@0"
I20260619 01:06:54.008013 5228 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 01:06:54.012759 5242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:54.014513 5228 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:06:54.014739 5228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "5743f431137e4198b84e005bedf110a1"
format_stamp: "Formatted at 2026-06-19 01:06:53 on dist-test-slave-5p8w"
server_key: "9ce6b0ef2d83a0364de953057f1d3e77"
server_key_iv: "e771000c795d7ebdc634d97765fbcb71"
server_key_version: "encryptionkey@0"
I20260619 01:06:54.015017 5228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:54.021723 5228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:54.022645 5228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:54.022950 5228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:54.024055 5228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:54.025707 5228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:54.025832 5228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:54.025980 5228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:54.026065 5228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:54.061003 5228 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35489
I20260619 01:06:54.061072 5354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35489 every 8 connection(s)
I20260619 01:06:54.062593 5228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:06:54.064018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 5228
I20260619 01:06:54.064216 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:06:54.064800 388 external_mini_cluster.cc:1428] Setting key b6cc9ac507a98a1c67c3792f5537145d
I20260619 01:06:54.068210 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42111
--builtin_ntp_servers=127.0.97.20:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 01:06:54.073943 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.065848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56691 (local address 127.0.97.62:42111)
0619 01:06:54.066089 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:54.066098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:54.068209 (+ 2111us) server_negotiation.cc:407] Connection header received
0619 01:06:54.071278 (+ 3069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:54.071282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:54.071362 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:54.071473 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:54.073202 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:54.073206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:54.073209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:54.073228 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:54.073259 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:54.073271 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:54.073444 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:54.073658 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 01:06:54.073714 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 01:06:54.075449 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.066631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42111 (local address 127.0.97.2:56691)
0619 01:06:54.068025 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:54.068083 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:54.070885 (+ 2802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:54.071787 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:54.071803 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:54.071972 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:54.072772 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:06:54.072778 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:54.072852 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:06:54.072985 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:54.073419 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:54.073433 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:54.073445 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 01:06:54.073745 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 01:06:54.074065 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":403,"threads_started":1}
I20260619 01:06:54.077152 5355 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42111
I20260619 01:06:54.077541 5355 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:54.078229 5355 heartbeater.cc:511] Master 127.0.97.62:42111 requested a full tablet report, sending...
I20260619 01:06:54.079893 5038 ts_manager.cc:194] Registered new tserver with Master: 5743f431137e4198b84e005bedf110a1 (127.0.97.2:35489)
W20260619 01:06:54.423043 5358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:54.423410 5358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:54.423573 5358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:54.431151 5358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:54.431326 5358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:54.431514 5358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:06:54.441138 5358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:54.442603 5358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:54.444536 5358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:54.459227 5364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:54.459290 5366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:54.459303 5363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:54.464545 5358 server_base.cc:1034] running on GCE node
I20260619 01:06:54.465255 5358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:54.466401 5358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:54.467638 5358 hybrid_clock.cc:648] HybridClock initialized: now 1781831214467586 us; error 51 us; skew 500 ppm
I20260619 01:06:54.468003 5358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:06:54.470530 5358 webserver.cc:460] Webserver started at http://127.0.97.3:46225/ using document root <none> and password file <none>
I20260619 01:06:54.471309 5358 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:54.471402 5358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:54.471776 5358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:54.475052 5358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "49409586395247a883b0421469fc0568"
format_stamp: "Formatted at 2026-06-19 01:06:54 on dist-test-slave-5p8w"
server_key: "ce069608295cf2aabec42f1b1eb64d0e"
server_key_iv: "522cdfdb708203ca99cf8b0c15aa368c"
server_key_version: "encryptionkey@0"
I20260619 01:06:54.475961 5358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "49409586395247a883b0421469fc0568"
format_stamp: "Formatted at 2026-06-19 01:06:54 on dist-test-slave-5p8w"
server_key: "ce069608295cf2aabec42f1b1eb64d0e"
server_key_iv: "522cdfdb708203ca99cf8b0c15aa368c"
server_key_version: "encryptionkey@0"
I20260619 01:06:54.481956 5358 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 01:06:54.486037 5372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:54.487859 5358 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:06:54.488092 5358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "49409586395247a883b0421469fc0568"
format_stamp: "Formatted at 2026-06-19 01:06:54 on dist-test-slave-5p8w"
server_key: "ce069608295cf2aabec42f1b1eb64d0e"
server_key_iv: "522cdfdb708203ca99cf8b0c15aa368c"
server_key_version: "encryptionkey@0"
I20260619 01:06:54.488443 5358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:54.494931 5358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:54.495814 5358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:54.496057 5358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:54.497139 5358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:06:54.498837 5358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:06:54.498965 5358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:54.499101 5358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:06:54.499187 5358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:54.534015 5358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37099
I20260619 01:06:54.534044 5484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37099 every 8 connection(s)
I20260619 01:06:54.535444 5358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:06:54.542532 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 5358
I20260619 01:06:54.542747 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:06:54.543326 388 external_mini_cluster.cc:1428] Setting key e42cbc220376d88094ee0531349c6724
I20260619 01:06:54.544642 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.538395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41287 (local address 127.0.97.62:42111)
0619 01:06:54.538607 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:54.538612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:54.540243 (+ 1631us) server_negotiation.cc:407] Connection header received
0619 01:06:54.542031 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:54.542035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:54.542092 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:54.542184 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:54.543807 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:54.543812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:54.543816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:54.543832 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:54.543894 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:54.543913 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:54.544055 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:54.544249 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 01:06:54.544333 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 01:06:54.545785 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.539013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42111 (local address 127.0.97.3:41287)
0619 01:06:54.540070 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:54.540115 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:54.541766 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:54.542502 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:54.542519 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:54.542662 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:54.543423 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:06:54.543427 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:54.543486 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:06:54.543611 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:54.544024 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:54.544032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:54.544038 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 01:06:54.544344 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 01:06:54.544630 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":331,"threads_started":1}
I20260619 01:06:54.547223 5485 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42111
I20260619 01:06:54.547556 5485 heartbeater.cc:464] Registering TS with master...
I20260619 01:06:54.548281 5485 heartbeater.cc:511] Master 127.0.97.62:42111 requested a full tablet report, sending...
I20260619 01:06:54.549963 5038 ts_manager.cc:194] Registered new tserver with Master: 49409586395247a883b0421469fc0568 (127.0.97.3:37099)
I20260619 01:06:54.559652 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:06:54.567431 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.564625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.0.97.62:42111)
0619 01:06:54.564822 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:54.564830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:54.565344 (+ 514us) server_negotiation.cc:407] Connection header received
0619 01:06:54.565584 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:54.565595 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:54.565653 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:54.565771 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:54.566760 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:54.566766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:54.566771 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:54.566796 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:54.566829 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:54.566841 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:54.566982 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:54.567081 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 01:06:54.567151 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:06:54.571745 5038 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:06:54.579069 5038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:06:54.601739 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.594227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35489 (local address 127.0.0.1:48152)
0619 01:06:54.595774 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:54.595814 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:54.595990 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:54.598298 (+ 2308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:54.598313 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:54.598458 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:54.599649 (+ 1191us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:06:54.599656 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:54.599702 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:06:54.599829 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:54.600607 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:54.600623 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:54.600636 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 01:06:54.600877 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 01:06:54.601036 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"thread_start_us":1279,"threads_started":1}
I20260619 01:06:54.602921 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.593540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.0.97.2:35489)
0619 01:06:54.594134 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:54.594145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:54.596152 (+ 2007us) server_negotiation.cc:407] Connection header received
0619 01:06:54.596241 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:54.596254 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:54.596605 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:54.596811 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:54.600153 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:54.600169 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:54.600194 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:54.600254 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:54.600341 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:54.600370 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:54.600539 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:54.602298 (+ 1759us) server_negotiation.cc:299] Negotiation successful
0619 01:06:54.602452 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":301,"threads_started":1}
I20260619 01:06:54.609640 5224 heartbeater.cc:503] Master 127.0.97.62:42111 was elected leader, sending a full tablet report...
I20260619 01:06:54.609517 5290 tablet_service.cc:1467] Processing CreateTablet for tablet df43cb9805a14a35ac53f71fd552ba32 (DEFAULT_TABLE table=test-table [id=625752efe20d4b09a2cc4501dee58895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:54.611507 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df43cb9805a14a35ac53f71fd552ba32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:54.613637 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.594847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37099 (local address 127.0.0.1:46772)
0619 01:06:54.603965 (+ 9118us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:54.603990 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:54.604114 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:54.611590 (+ 7476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:54.611596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:54.611649 (+ 53us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:54.612364 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:06:54.612367 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:54.612392 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:06:54.612456 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:54.613063 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:54.613080 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:54.613116 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 01:06:54.613227 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 01:06:54.613318 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8925,"thread_start_us":13590,"threads_started":1}
I20260619 01:06:54.614588 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.594904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46772 (local address 127.0.97.3:37099)
0619 01:06:54.609664 (+ 14760us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:54.609677 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:54.609718 (+ 41us) server_negotiation.cc:407] Connection header received
0619 01:06:54.609806 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:54.609816 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:54.610117 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:54.610305 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:54.612656 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:54.612666 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:54.612683 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:54.612711 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:54.612776 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:54.612796 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:54.612963 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:54.614048 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0619 01:06:54.614161 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14554,"thread_start_us":15101,"threads_started":1}
I20260619 01:06:54.619225 5420 tablet_service.cc:1467] Processing CreateTablet for tablet df43cb9805a14a35ac53f71fd552ba32 (DEFAULT_TABLE table=test-table [id=625752efe20d4b09a2cc4501dee58895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:54.621599 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.595203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43287 (local address 127.0.0.1:47460)
0619 01:06:54.608129 (+ 12926us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:54.608153 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:54.608301 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:54.619440 (+ 11139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:54.619446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:54.619481 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:54.620275 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:06:54.620279 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:54.620299 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:06:54.620350 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:54.620889 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:54.620895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:54.620901 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 01:06:54.621032 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 01:06:54.621108 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12737,"thread_start_us":24052,"threads_started":1}
I20260619 01:06:54.621738 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df43cb9805a14a35ac53f71fd552ba32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:54.622467 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.595956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.0.97.1:43287)
0619 01:06:54.617849 (+ 21893us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:54.617860 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:54.617895 (+ 35us) server_negotiation.cc:407] Connection header received
0619 01:06:54.617969 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:54.617979 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:54.618248 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:54.618433 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:54.620639 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:54.620650 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:54.620669 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:54.620689 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:54.620753 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:54.620773 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:54.621913 (+ 1140us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:54.621988 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 01:06:54.622109 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21707,"thread_start_us":21556,"threads_started":1}
I20260619 01:06:54.627836 5503 tablet_bootstrap.cc:492] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: Bootstrap starting.
I20260619 01:06:54.629937 5159 tablet_service.cc:1467] Processing CreateTablet for tablet df43cb9805a14a35ac53f71fd552ba32 (DEFAULT_TABLE table=test-table [id=625752efe20d4b09a2cc4501dee58895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:06:54.631443 5503 tablet_bootstrap.cc:654] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:54.631567 5159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df43cb9805a14a35ac53f71fd552ba32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:54.632797 5503 log.cc:824] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:54.636186 5503 tablet_bootstrap.cc:492] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: No bootstrap required, opened a new log
I20260619 01:06:54.636528 5503 ts_tablet_manager.cc:1397] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 01:06:54.637770 5505 tablet_bootstrap.cc:492] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: Bootstrap starting.
I20260619 01:06:54.642875 5505 tablet_bootstrap.cc:654] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:54.644315 5505 log.cc:824] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:54.646765 5507 tablet_bootstrap.cc:492] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27: Bootstrap starting.
I20260619 01:06:54.647567 5503 raft_consensus.cc:348] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 } } peers { permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 } } peers { permanent_uuid: "7fb545544c8a4785aa96c87654292b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } }
I20260619 01:06:54.648046 5505 tablet_bootstrap.cc:492] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: No bootstrap required, opened a new log
I20260619 01:06:54.648106 5503 raft_consensus.cc:374] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:54.648211 5503 raft_consensus.cc:729] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5743f431137e4198b84e005bedf110a1, State: Initialized, Role: FOLLOWER
I20260619 01:06:54.648361 5505 ts_tablet_manager.cc:1397] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260619 01:06:54.648962 5503 consensus_queue.cc:260] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 } } peers { permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 } } peers { permanent_uuid: "7fb545544c8a4785aa96c87654292b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } }
I20260619 01:06:54.651003 5507 tablet_bootstrap.cc:654] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:54.652210 5507 log.cc:824] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:54.658536 5355 heartbeater.cc:503] Master 127.0.97.62:42111 was elected leader, sending a full tablet report...
I20260619 01:06:54.659399 5503 ts_tablet_manager.cc:1428] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: Time spent starting tablet: real 0.023s user 0.015s sys 0.008s
I20260619 01:06:54.660992 5507 tablet_bootstrap.cc:492] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27: No bootstrap required, opened a new log
I20260619 01:06:54.661404 5507 ts_tablet_manager.cc:1397] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20260619 01:06:54.662302 5505 raft_consensus.cc:348] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 } } peers { permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 } } peers { permanent_uuid: "7fb545544c8a4785aa96c87654292b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } }
I20260619 01:06:54.662860 5505 raft_consensus.cc:374] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:54.662972 5505 raft_consensus.cc:729] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49409586395247a883b0421469fc0568, State: Initialized, Role: FOLLOWER
I20260619 01:06:54.663759 5505 consensus_queue.cc:260] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 } } peers { permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 } } peers { permanent_uuid: "7fb545544c8a4785aa96c87654292b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } }
I20260619 01:06:54.665966 5485 heartbeater.cc:503] Master 127.0.97.62:42111 was elected leader, sending a full tablet report...
I20260619 01:06:54.668226 5505 ts_tablet_manager.cc:1428] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
I20260619 01:06:54.670835 5507 raft_consensus.cc:348] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 } } peers { permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 } } peers { permanent_uuid: "7fb545544c8a4785aa96c87654292b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } }
I20260619 01:06:54.671306 5507 raft_consensus.cc:374] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:54.671419 5507 raft_consensus.cc:729] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb545544c8a4785aa96c87654292b27, State: Initialized, Role: FOLLOWER
I20260619 01:06:54.672027 5507 consensus_queue.cc:260] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 } } peers { permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 } } peers { permanent_uuid: "7fb545544c8a4785aa96c87654292b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } }
I20260619 01:06:54.673858 5507 ts_tablet_manager.cc:1428] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
W20260619 01:06:54.787290 5486 tablet.cc:2369] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:06:54.814977 5356 tablet.cc:2369] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:06:54.843032 5225 tablet.cc:2369] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:06:54.905037 5511 raft_consensus.cc:482] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:54.905426 5511 raft_consensus.cc:504] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 } } peers { permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 } } peers { permanent_uuid: "7fb545544c8a4785aa96c87654292b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } }
I20260619 01:06:54.907652 5511 leader_election.cc:288] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5743f431137e4198b84e005bedf110a1 (127.0.97.2:35489), 49409586395247a883b0421469fc0568 (127.0.97.3:37099)
I20260619 01:06:54.910768 5508 raft_consensus.cc:482] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:06:54.911219 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.908243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36169 (local address 127.0.97.3:37099)
0619 01:06:54.908498 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:54.908506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:54.908950 (+ 444us) server_negotiation.cc:407] Connection header received
0619 01:06:54.909147 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:54.909157 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:54.909232 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:54.909352 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:54.910497 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:54.910503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:54.910507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:54.910526 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:54.910563 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:54.910579 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:54.910726 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:54.910897 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 01:06:54.910958 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 01:06:54.911237 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.908058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36523 (local address 127.0.97.2:35489)
0619 01:06:54.908283 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:54.908293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:54.909219 (+ 926us) server_negotiation.cc:407] Connection header received
0619 01:06:54.909368 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:54.909373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:54.909428 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:54.909523 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:54.910541 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:54.910545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:54.910549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:54.910563 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:54.910616 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:54.910632 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:54.910756 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:54.910865 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 01:06:54.910930 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 01:06:54.911386 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.908005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35489 (local address 127.0.97.1:36523)
0619 01:06:54.909120 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:54.909134 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:54.909224 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:54.909570 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:54.909576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:54.909598 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:54.910318 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:06:54.910321 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:54.910333 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:06:54.910364 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:54.910741 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:54.910748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:54.910754 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 01:06:54.910925 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 01:06:54.910976 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"thread_start_us":784,"threads_started":1}
I20260619 01:06:54.911921 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.907955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37099 (local address 127.0.97.1:36169)
0619 01:06:54.908813 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:54.908832 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:54.909013 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:54.909391 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:54.909395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:54.909419 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:54.910240 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:06:54.910245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:54.910266 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:06:54.910322 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:54.910767 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:54.910771 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:54.910775 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 01:06:54.911475 (+ 700us) client_negotiation.cc:240] Negotiation successful
0619 01:06:54.911533 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":535,"threads_started":1}
I20260619 01:06:54.911237 5508 raft_consensus.cc:504] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 } } peers { permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 } } peers { permanent_uuid: "7fb545544c8a4785aa96c87654292b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } }
I20260619 01:06:54.912640 5310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df43cb9805a14a35ac53f71fd552ba32" candidate_uuid: "7fb545544c8a4785aa96c87654292b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5743f431137e4198b84e005bedf110a1" is_pre_election: true
I20260619 01:06:54.912884 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df43cb9805a14a35ac53f71fd552ba32" candidate_uuid: "7fb545544c8a4785aa96c87654292b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49409586395247a883b0421469fc0568" is_pre_election: true
I20260619 01:06:54.913180 5310 raft_consensus.cc:2454] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb545544c8a4785aa96c87654292b27 in term 0.
I20260619 01:06:54.913458 5440 raft_consensus.cc:2454] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb545544c8a4785aa96c87654292b27 in term 0.
I20260619 01:06:54.913964 5112 leader_election.cc:302] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5743f431137e4198b84e005bedf110a1, 7fb545544c8a4785aa96c87654292b27; no voters:
I20260619 01:06:54.914389 5511 raft_consensus.cc:2781] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:06:54.914563 5511 raft_consensus.cc:482] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:06:54.914673 5511 raft_consensus.cc:3037] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:54.918839 5508 leader_election.cc:288] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49409586395247a883b0421469fc0568 (127.0.97.3:37099), 7fb545544c8a4785aa96c87654292b27 (127.0.97.1:43287)
I20260619 01:06:54.919021 5511 raft_consensus.cc:504] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 } } peers { permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 } } peers { permanent_uuid: "7fb545544c8a4785aa96c87654292b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } }
I20260619 01:06:54.920401 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df43cb9805a14a35ac53f71fd552ba32" candidate_uuid: "7fb545544c8a4785aa96c87654292b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49409586395247a883b0421469fc0568"
I20260619 01:06:54.920624 5440 raft_consensus.cc:3037] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:54.920785 5310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df43cb9805a14a35ac53f71fd552ba32" candidate_uuid: "7fb545544c8a4785aa96c87654292b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5743f431137e4198b84e005bedf110a1"
I20260619 01:06:54.920994 5310 raft_consensus.cc:3037] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:54.921734 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.914951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49647 (local address 127.0.97.3:37099)
0619 01:06:54.915118 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:54.915123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:54.919121 (+ 3998us) server_negotiation.cc:407] Connection header received
0619 01:06:54.919422 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:54.919428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:54.919500 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:54.919557 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:54.920795 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:54.920803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:54.920810 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:54.920834 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:54.920875 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:54.920893 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:54.921010 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:54.921316 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 01:06:54.921398 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:06:54.922219 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.915106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57865 (local address 127.0.97.1:43287)
0619 01:06:54.919833 (+ 4727us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:54.919839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:54.919857 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:06:54.919905 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:54.919912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:54.919970 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:54.920066 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:54.921505 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:54.921511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:54.921516 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:54.921536 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:54.921571 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:54.921585 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:54.921674 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:54.921923 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 01:06:54.921982 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4584}
I20260619 01:06:54.922616 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.914608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37099 (local address 127.0.97.2:49647)
0619 01:06:54.918961 (+ 4353us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:54.918990 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:54.919234 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:54.919769 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:54.919773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:54.919796 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:54.920536 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:06:54.920541 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:54.920562 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:06:54.920618 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:54.921106 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:54.921113 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:54.921118 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 01:06:54.922043 (+ 925us) client_negotiation.cc:240] Negotiation successful
0619 01:06:54.922099 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4161,"thread_start_us":4517,"threads_started":1}
I20260619 01:06:54.923048 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.914745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43287 (local address 127.0.97.2:57865)
0619 01:06:54.919449 (+ 4704us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:54.919468 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:54.919575 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:54.920113 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:54.920121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:54.920486 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:06:54.921252 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:06:54.921261 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:06:54.921280 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:06:54.921353 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:06:54.921768 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:06:54.921773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:06:54.921778 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 01:06:54.922477 (+ 699us) client_negotiation.cc:240] Negotiation successful
0619 01:06:54.922532 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4584,"thread_start_us":3770,"threads_started":1}
I20260619 01:06:54.923218 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df43cb9805a14a35ac53f71fd552ba32" candidate_uuid: "5743f431137e4198b84e005bedf110a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49409586395247a883b0421469fc0568" is_pre_election: true
I20260619 01:06:54.923920 5179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df43cb9805a14a35ac53f71fd552ba32" candidate_uuid: "5743f431137e4198b84e005bedf110a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb545544c8a4785aa96c87654292b27" is_pre_election: true
I20260619 01:06:54.924439 5179 raft_consensus.cc:2379] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5743f431137e4198b84e005bedf110a1 in current term 1: Already voted for candidate 7fb545544c8a4785aa96c87654292b27 in this term.
I20260619 01:06:54.925117 5440 raft_consensus.cc:2454] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb545544c8a4785aa96c87654292b27 in term 1.
I20260619 01:06:54.925314 5245 leader_election.cc:302] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5743f431137e4198b84e005bedf110a1; no voters: 49409586395247a883b0421469fc0568, 7fb545544c8a4785aa96c87654292b27
I20260619 01:06:54.925540 5113 leader_election.cc:302] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49409586395247a883b0421469fc0568, 7fb545544c8a4785aa96c87654292b27; no voters:
I20260619 01:06:54.925534 5511 leader_election.cc:288] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [CANDIDATE]: Term 1 election: Requested vote from peers 5743f431137e4198b84e005bedf110a1 (127.0.97.2:35489), 49409586395247a883b0421469fc0568 (127.0.97.3:37099)
I20260619 01:06:54.926512 5310 raft_consensus.cc:2454] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb545544c8a4785aa96c87654292b27 in term 1.
I20260619 01:06:54.926687 5511 raft_consensus.cc:2781] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:54.926802 5508 raft_consensus.cc:2726] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 01:06:54.926846 5511 raft_consensus.cc:686] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 1 LEADER]: Becoming Leader. State: Replica: 7fb545544c8a4785aa96c87654292b27, State: Running, Role: LEADER
I20260619 01:06:54.927455 5511 consensus_queue.cc:237] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 } } peers { permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 } } peers { permanent_uuid: "7fb545544c8a4785aa96c87654292b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } }
I20260619 01:06:54.932783 5037 catalog_manager.cc:5495] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fb545544c8a4785aa96c87654292b27 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7fb545544c8a4785aa96c87654292b27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb545544c8a4785aa96c87654292b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43287 } health_report { overall_health: HEALTHY } } }
I20260619 01:06:54.988512 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:54.985875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.0.97.1:43287)
0619 01:06:54.986088 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:54.986095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:54.986216 (+ 121us) server_negotiation.cc:407] Connection header received
0619 01:06:54.986505 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:54.986510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:54.986613 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:54.986743 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:54.987775 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:54.987781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:54.987786 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:54.987814 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:54.987848 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:54.987861 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:54.988020 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:54.988116 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 01:06:54.988181 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 01:06:54.997915 5440 raft_consensus.cc:1261] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568 [term 1 FOLLOWER]: Refusing update from remote peer 7fb545544c8a4785aa96c87654292b27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:54.997973 5310 raft_consensus.cc:1261] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 1 FOLLOWER]: Refusing update from remote peer 7fb545544c8a4785aa96c87654292b27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:06:54.998838 5516 consensus_queue.cc:1035] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49409586395247a883b0421469fc0568" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:54.999207 5511 consensus_queue.cc:1035] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5743f431137e4198b84e005bedf110a1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:06:55.030812 5037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:55.031132 5037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:06:55.035615 5037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dd127638c9d456182d26d708d1f3c30: Sending DeleteTablet for 3 replicas of tablet df43cb9805a14a35ac53f71fd552ba32
I20260619 01:06:55.037084 5290 tablet_service.cc:1514] Processing DeleteTablet for tablet df43cb9805a14a35ac53f71fd552ba32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:55 UTC) from {username='slave'} at 127.0.0.1:48152
I20260619 01:06:55.037091 5159 tablet_service.cc:1514] Processing DeleteTablet for tablet df43cb9805a14a35ac53f71fd552ba32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:55 UTC) from {username='slave'} at 127.0.0.1:47460
I20260619 01:06:55.037967 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet df43cb9805a14a35ac53f71fd552ba32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:06:55 UTC) from {username='slave'} at 127.0.0.1:46772
I20260619 01:06:55.043869 5530 tablet_replica.cc:330] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: stopping tablet replica
I20260619 01:06:55.044377 5530 raft_consensus.cc:2229] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:55.044920 5530 raft_consensus.cc:2258] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:55.045007 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 5097
I20260619 01:06:55.053489 5530 ts_tablet_manager.cc:1905] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:55.062871 5529 tablet_replica.cc:330] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: stopping tablet replica
I20260619 01:06:55.063690 5529 raft_consensus.cc:2229] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:06:55.064420 5529 raft_consensus.cc:2258] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:55.068135 5528 tablet_replica.cc:330] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27: stopping tablet replica
I20260619 01:06:55.068182 5529 ts_tablet_manager.cc:1905] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:06:55.068686 5530 ts_tablet_manager.cc:1918] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:06:55.069280 5528 raft_consensus.cc:2229] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:06:55.069593 5530 log.cc:1196] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/df43cb9805a14a35ac53f71fd552ba32
I20260619 01:06:55.069567 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:55.047030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.0.97.1:43287)
0619 01:06:55.047300 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:55.047310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:55.058157 (+ 10847us) server_negotiation.cc:407] Connection header received
0619 01:06:55.058841 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:55.058852 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:55.058956 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:55.059060 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:55.068595 (+ 9535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:55.068602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:55.068608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:55.068637 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:55.068683 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:55.068701 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:55.068809 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:55.069078 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 01:06:55.069151 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 01:06:55.070329 5530 ts_tablet_manager.cc:1939] T df43cb9805a14a35ac53f71fd552ba32 P 49409586395247a883b0421469fc0568: Deleting consensus metadata
I20260619 01:06:55.070731 5219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:55.072888 5023 catalog_manager.cc:4841] TS 49409586395247a883b0421469fc0568 (127.0.97.3:37099): tablet df43cb9805a14a35ac53f71fd552ba32 (table test-table [id=625752efe20d4b09a2cc4501dee58895]) successfully deleted
I20260619 01:06:55.077075 5529 ts_tablet_manager.cc:1918] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:06:55.077349 5529 log.cc:1196] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/df43cb9805a14a35ac53f71fd552ba32
I20260619 01:06:55.077919 5529 ts_tablet_manager.cc:1939] T df43cb9805a14a35ac53f71fd552ba32 P 5743f431137e4198b84e005bedf110a1: Deleting consensus metadata
I20260619 01:06:55.079656 5022 catalog_manager.cc:4841] TS 5743f431137e4198b84e005bedf110a1 (127.0.97.2:35489): tablet df43cb9805a14a35ac53f71fd552ba32 (table test-table [id=625752efe20d4b09a2cc4501dee58895]) successfully deleted
I20260619 01:06:55.186326 5528 raft_consensus.cc:2258] T df43cb9805a14a35ac53f71fd552ba32 P 7fb545544c8a4785aa96c87654292b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:06:55.186771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 5097
I20260619 01:06:55.204869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 5228
W20260619 01:06:55.204877 5024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43287 (error 108)
I20260619 01:06:55.207206 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:55.206249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43287 (local address 127.0.0.1:47488)
0619 01:06:55.206769 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:55.206964 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":278,"threads_started":1}
W20260619 01:06:55.207489 5024 catalog_manager.cc:4584] TS 7fb545544c8a4785aa96c87654292b27 (127.0.97.1:43287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df43cb9805a14a35ac53f71fd552ba32: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43287: connect: Connection refused (error 111)
I20260619 01:06:55.208192 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:55.205940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.0.97.2:35489)
0619 01:06:55.206119 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:55.206127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:55.206150 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:06:55.206217 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:55.206223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:55.206333 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:55.206471 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:55.207501 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:55.207522 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:55.207525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:55.207541 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:55.207572 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:55.207585 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:55.207745 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:55.207865 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 01:06:55.207911 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 01:06:55.208941 5350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:55.261181 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:55.260576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43287 (local address 127.0.0.1:47498)
0619 01:06:55.260841 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:55.260980 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 01:06:55.262375 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:55.261924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43287 (local address 127.0.0.1:47508)
0619 01:06:55.262102 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:55.262198 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 01:06:55.302296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 5228
I20260619 01:06:55.320206 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 5358
I20260619 01:06:55.322974 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:55.322542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43287 (local address 127.0.0.1:47524)
0619 01:06:55.322729 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:55.322835 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 01:06:55.323522 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:55.321072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.0.97.3:37099)
0619 01:06:55.321267 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:55.321272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:55.321348 (+ 76us) server_negotiation.cc:407] Connection header received
0619 01:06:55.321564 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:55.321568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:55.321647 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:55.321766 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:55.322773 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:55.322777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:55.322781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:55.322796 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:55.322828 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:55.322841 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:55.322980 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:55.323150 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 01:06:55.323216 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 01:06:55.324215 5480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:55.324493 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:55.324072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43287 (local address 127.0.0.1:47526)
0619 01:06:55.324232 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:55.324310 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 01:06:55.405217 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:55.404747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43287 (local address 127.0.0.1:47534)
0619 01:06:55.404977 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:55.405070 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 01:06:55.406394 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:55.405977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43287 (local address 127.0.0.1:47538)
0619 01:06:55.406161 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:55.406232 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260619 01:06:55.415828 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 5358
I20260619 01:06:55.432370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 5007
I20260619 01:06:55.433581 5068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:55.539986 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 5007
2026-06-19T01:06:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3029 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 01:06:55.582068 388 mini_oidc.cc:152] Starting JWKS server
I20260619 01:06:55.584703 388 webserver.cc:460] Webserver started at https://127.0.0.1:40089/ using document root <none> and password file <none>
I20260619 01:06:55.585052 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 01:06:55.585724 388 webserver.cc:460] Webserver started at http://127.0.0.1:46505/ using document root <none> and password file <none>
2026-06-19T01:06:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:55Z Disabled control of system clock
I20260619 01:06:55.614566 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35529
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35529
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40089/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/testchainca.cert with env {}
W20260619 01:06:55.961230 5543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:55.961597 5543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:55.961688 5543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:55.969220 5543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:55.969386 5543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:55.969426 5543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:55.969476 5543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:55.969520 5543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 01:06:55.969583 5543 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 01:06:55.969621 5543 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40089/jwks.json
I20260619 01:06:55.980057 5543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40089/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:55.981431 5543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:55.982731 5543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:55.995260 5548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:55.995219 5549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:55.995265 5551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:55.996376 5543 server_base.cc:1034] running on GCE node
I20260619 01:06:55.997350 5543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:55.998911 5543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:56.000160 5543 hybrid_clock.cc:648] HybridClock initialized: now 1781831216000110 us; error 44 us; skew 500 ppm
I20260619 01:06:56.000562 5543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:06:56.003180 5543 webserver.cc:460] Webserver started at http://127.0.97.62:46017/ using document root <none> and password file <none>
I20260619 01:06:56.003953 5543 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:56.004098 5543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:56.004433 5543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:56.007673 5543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "d02d48e698cb4d9d94d6ca9137655afd"
format_stamp: "Formatted at 2026-06-19 01:06:55 on dist-test-slave-5p8w"
server_key: "d7c975506b775b0db37101bcecd624b3"
server_key_iv: "49e236a166001fdab01f3a68e4947fe7"
server_key_version: "encryptionkey@0"
I20260619 01:06:56.008635 5543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "d02d48e698cb4d9d94d6ca9137655afd"
format_stamp: "Formatted at 2026-06-19 01:06:55 on dist-test-slave-5p8w"
server_key: "d7c975506b775b0db37101bcecd624b3"
server_key_iv: "49e236a166001fdab01f3a68e4947fe7"
server_key_version: "encryptionkey@0"
I20260619 01:06:56.014416 5543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 01:06:56.018421 5557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:56.019929 5543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260619 01:06:56.020146 5543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "d02d48e698cb4d9d94d6ca9137655afd"
format_stamp: "Formatted at 2026-06-19 01:06:55 on dist-test-slave-5p8w"
server_key: "d7c975506b775b0db37101bcecd624b3"
server_key_iv: "49e236a166001fdab01f3a68e4947fe7"
server_key_version: "encryptionkey@0"
I20260619 01:06:56.020396 5543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:56.051635 5543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:56.052598 5543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:56.052918 5543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:56.072316 5543 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35529
I20260619 01:06:56.072337 5608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35529 every 8 connection(s)
I20260619 01:06:56.073962 5543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:56.076200 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 5543
I20260619 01:06:56.076395 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:56.077190 388 external_mini_cluster.cc:1428] Setting key fde35f7a415d7127995b2b96c6fc0e99
I20260619 01:06:56.078920 5609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:56.091409 5609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd: Bootstrap starting.
I20260619 01:06:56.091452 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:56.079263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.0.97.62:35529)
0619 01:06:56.080260 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:56.080272 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:56.080316 (+ 44us) server_negotiation.cc:407] Connection header received
0619 01:06:56.082014 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:56.082039 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:56.082575 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:56.083051 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:56.083711 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:56.085126 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:56.086687 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:56.087425 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:56.087633 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:56.088950 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:56.088958 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:56.088978 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:56.089655 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:56.089734 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:56.089768 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:56.089895 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:56.090133 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 01:06:56.090457 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":329,"threads_started":1}
I20260619 01:06:56.095254 5609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:56.095602 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:06:56.096370 5609 log.cc:824] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:56.099562 5609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd: No bootstrap required, opened a new log
I20260619 01:06:56.107280 5609 raft_consensus.cc:348] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02d48e698cb4d9d94d6ca9137655afd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35529 } }
I20260619 01:06:56.107695 5609 raft_consensus.cc:374] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:56.107796 5609 raft_consensus.cc:729] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02d48e698cb4d9d94d6ca9137655afd, State: Initialized, Role: FOLLOWER
I20260619 01:06:56.108492 5609 consensus_queue.cc:260] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02d48e698cb4d9d94d6ca9137655afd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35529 } }
I20260619 01:06:56.108748 5609 raft_consensus.cc:388] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:56.108891 5609 raft_consensus.cc:482] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:56.109069 5609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:56.111550 5609 raft_consensus.cc:504] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02d48e698cb4d9d94d6ca9137655afd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35529 } }
I20260619 01:06:56.112177 5609 leader_election.cc:302] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d02d48e698cb4d9d94d6ca9137655afd; no voters:
I20260619 01:06:56.112771 5609 leader_election.cc:288] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:56.112919 5614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:56.113662 5614 raft_consensus.cc:686] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [term 1 LEADER]: Becoming Leader. State: Replica: d02d48e698cb4d9d94d6ca9137655afd, State: Running, Role: LEADER
I20260619 01:06:56.114248 5614 consensus_queue.cc:237] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02d48e698cb4d9d94d6ca9137655afd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35529 } }
I20260619 01:06:56.115060 5609 sys_catalog.cc:564] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:56.117102 5615 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d02d48e698cb4d9d94d6ca9137655afd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02d48e698cb4d9d94d6ca9137655afd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35529 } } }
I20260619 01:06:56.117487 5615 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:56.117493 5616 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d02d48e698cb4d9d94d6ca9137655afd. Latest consensus state: current_term: 1 leader_uuid: "d02d48e698cb4d9d94d6ca9137655afd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02d48e698cb4d9d94d6ca9137655afd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35529 } } }
I20260619 01:06:56.117901 5616 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:56.118528 5620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:56.125233 5620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:56.143785 5620 catalog_manager.cc:1347] Generated new cluster ID: b01b687caee241769f05af0ac1a73842
I20260619 01:06:56.143927 5620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:56.170425 5620 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:56.171871 5620 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:56.182822 5620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d02d48e698cb4d9d94d6ca9137655afd: Generated new TSK 0
I20260619 01:06:56.183750 5620 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 01:06:56.226046 5639 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 01:06:56.226235 5639 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 01:06:56.227470 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:56.204501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.0.97.62:35529)
0619 01:06:56.204709 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:56.204716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:56.205156 (+ 440us) server_negotiation.cc:407] Connection header received
0619 01:06:56.205535 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:56.205541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:56.205641 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:56.205815 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0619 01:06:56.206354 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:56.207523 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:56.208962 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:56.209726 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:56.209869 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:56.210448 (+ 579us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 01:06:56.226716 (+ 16268us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 01:06:56.227130 (+ 414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 01:06:56.227602 5612 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 01:06:56.228855 5638 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:06:56.204322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35529 (local address 127.0.0.1:43762)
0619 01:06:56.205029 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:56.205049 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:56.205211 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:56.205860 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:56.205863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:56.205873 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 01:06:56.206198 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:56.206206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:56.207663 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:56.207666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:56.208821 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:56.208832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:56.209861 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:56.209922 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:56.210058 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:56.210126 (+ 68us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 01:06:56.227052 (+ 16926us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 01:06:56.227346 (+ 294us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35529: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":369,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:35529: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 01:06:56.230808 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 5543
I20260619 01:06:56.232086 5604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:56.328342 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 5543
2026-06-19T01:06:56Z chronyd exiting
I20260619 01:06:56.380152 388 test_util.cc:175] -----------------------------------------------
I20260619 01:06:56.380311 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781831171286417-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (802 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 01:06:56.384196 388 mini_oidc.cc:152] Starting JWKS server
I20260619 01:06:56.386143 388 webserver.cc:460] Webserver started at https://127.0.0.1:42753/ using document root <none> and password file <none>
I20260619 01:06:56.386413 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 01:06:56.387072 388 webserver.cc:460] Webserver started at http://127.0.0.1:40097/ using document root <none> and password file <none>
2026-06-19T01:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:56Z Disabled control of system clock
I20260619 01:06:56.415022 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38085
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38085
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42753/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/testchainca.cert with env {}
W20260619 01:06:56.759812 5646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:56.760141 5646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:56.760319 5646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:56.767867 5646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:56.768009 5646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:56.768049 5646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:56.768165 5646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:56.768249 5646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 01:06:56.768342 5646 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 01:06:56.768421 5646 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42753/jwks.json
I20260619 01:06:56.778704 5646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42753/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:56.780154 5646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:56.781564 5646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:56.794257 5654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:56.794318 5652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:56.794348 5651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:56.795079 5646 server_base.cc:1034] running on GCE node
I20260619 01:06:56.795928 5646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:56.797528 5646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:56.798744 5646 hybrid_clock.cc:648] HybridClock initialized: now 1781831216798694 us; error 41 us; skew 500 ppm
I20260619 01:06:56.799161 5646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:06:56.801659 5646 webserver.cc:460] Webserver started at http://127.0.97.62:34479/ using document root <none> and password file <none>
I20260619 01:06:56.802397 5646 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:56.802516 5646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:56.802884 5646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:56.806077 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "ac0ebd68b59b4da883046a456f2f4cc4"
format_stamp: "Formatted at 2026-06-19 01:06:56 on dist-test-slave-5p8w"
server_key: "6e9e03bf0df44875c72c369e9d17c95e"
server_key_iv: "e9de39edb1d0f0ddaefabc507ed64bd8"
server_key_version: "encryptionkey@0"
I20260619 01:06:56.806943 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "ac0ebd68b59b4da883046a456f2f4cc4"
format_stamp: "Formatted at 2026-06-19 01:06:56 on dist-test-slave-5p8w"
server_key: "6e9e03bf0df44875c72c369e9d17c95e"
server_key_iv: "e9de39edb1d0f0ddaefabc507ed64bd8"
server_key_version: "encryptionkey@0"
I20260619 01:06:56.812448 5646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 01:06:56.816360 5660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:56.818111 5646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:06:56.818382 5646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "ac0ebd68b59b4da883046a456f2f4cc4"
format_stamp: "Formatted at 2026-06-19 01:06:56 on dist-test-slave-5p8w"
server_key: "6e9e03bf0df44875c72c369e9d17c95e"
server_key_iv: "e9de39edb1d0f0ddaefabc507ed64bd8"
server_key_version: "encryptionkey@0"
I20260619 01:06:56.818672 5646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:56.854790 5646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:56.855772 5646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:56.856097 5646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:56.875397 5646 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38085
I20260619 01:06:56.875407 5711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38085 every 8 connection(s)
I20260619 01:06:56.876883 5646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:56.881126 5712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:56.886878 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 5646
I20260619 01:06:56.887090 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:56.887872 388 external_mini_cluster.cc:1428] Setting key 44b4299527de625fed061cb4b73de374
I20260619 01:06:56.893739 5712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4: Bootstrap starting.
I20260619 01:06:56.897152 5712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:56.898334 5712 log.cc:824] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:56.901809 5712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4: No bootstrap required, opened a new log
I20260619 01:06:56.902628 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:56.889995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.0.97.62:38085)
0619 01:06:56.891178 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:56.891191 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:56.891242 (+ 51us) server_negotiation.cc:407] Connection header received
0619 01:06:56.893036 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:56.893061 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:56.893711 (+ 650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:56.894279 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:56.894884 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:56.896213 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:56.897698 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:56.898450 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:56.898646 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:56.900093 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:56.900100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:56.900117 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:56.900698 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:56.900779 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:56.900802 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:56.900901 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:56.901160 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 01:06:56.901546 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":382,"threads_started":1}
I20260619 01:06:56.906703 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:06:56.910982 5712 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0ebd68b59b4da883046a456f2f4cc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38085 } }
I20260619 01:06:56.911367 5712 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:56.911487 5712 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac0ebd68b59b4da883046a456f2f4cc4, State: Initialized, Role: FOLLOWER
I20260619 01:06:56.912128 5712 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0ebd68b59b4da883046a456f2f4cc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38085 } }
I20260619 01:06:56.912362 5712 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:56.912544 5712 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:56.912705 5712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:56.915069 5712 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0ebd68b59b4da883046a456f2f4cc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38085 } }
I20260619 01:06:56.915652 5712 leader_election.cc:302] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac0ebd68b59b4da883046a456f2f4cc4; no voters:
I20260619 01:06:56.916395 5712 leader_election.cc:288] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:56.916570 5717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:56.917400 5717 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [term 1 LEADER]: Becoming Leader. State: Replica: ac0ebd68b59b4da883046a456f2f4cc4, State: Running, Role: LEADER
I20260619 01:06:56.917981 5717 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0ebd68b59b4da883046a456f2f4cc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38085 } }
I20260619 01:06:56.918957 5712 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:56.921041 5719 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac0ebd68b59b4da883046a456f2f4cc4. Latest consensus state: current_term: 1 leader_uuid: "ac0ebd68b59b4da883046a456f2f4cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0ebd68b59b4da883046a456f2f4cc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38085 } } }
I20260619 01:06:56.921532 5719 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:56.922034 5718 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac0ebd68b59b4da883046a456f2f4cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac0ebd68b59b4da883046a456f2f4cc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38085 } } }
I20260619 01:06:56.922395 5718 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4 [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:56.923417 5725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:56.929374 5725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:56.938558 5725 catalog_manager.cc:1347] Generated new cluster ID: e07a7e3086fa4766804e106d4bcfb45d
I20260619 01:06:56.938690 5725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:56.958987 5725 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:56.960522 5725 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:56.973002 5725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac0ebd68b59b4da883046a456f2f4cc4: Generated new TSK 0
I20260619 01:06:56.974270 5725 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 01:06:57.016773 5742 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 01:06:57.016921 5742 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 01:06:57.018288 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:56.997427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.0.97.62:38085)
0619 01:06:56.997627 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:56.997635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:56.998109 (+ 474us) server_negotiation.cc:407] Connection header received
0619 01:06:56.998365 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:56.998371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:56.998470 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:56.998660 (+ 190us) server_negotiation.cc:226] Negotiated authn=JWT
0619 01:06:56.999117 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:57.000298 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:57.001807 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:57.002517 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:57.002651 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:57.003186 (+ 535us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 01:06:57.017551 (+ 14365us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42753/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 01:06:57.017969 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42753/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 01:06:57.018334 5741 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:06:56.997321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38085 (local address 127.0.0.1:41044)
0619 01:06:56.998003 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:56.998020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:56.998149 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:56.998673 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:56.998676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:56.998685 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 01:06:56.998952 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:56.998960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:57.000466 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:57.000471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:57.001657 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:57.001670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:57.002653 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:57.002699 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:57.002843 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:57.002880 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 01:06:57.017827 (+ 14947us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42753/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 01:06:57.018041 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:38085: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42753/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":390,"threads_started":1}
W20260619 01:06:57.018425 5715 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41044: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42753/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 01:06:57.019593 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 5646
I20260619 01:06:57.020975 5707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:57.116379 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 5646
2026-06-19T01:06:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (792 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 01:06:57.177430 388 mini_oidc.cc:152] Starting JWKS server
I20260619 01:06:57.179360 388 webserver.cc:460] Webserver started at https://127.0.0.1:37825/ using document root <none> and password file <none>
I20260619 01:06:57.179622 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 01:06:57.180302 388 webserver.cc:460] Webserver started at http://127.0.0.1:33763/ using document root <none> and password file <none>
2026-06-19T01:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:57Z Disabled control of system clock
I20260619 01:06:57.209180 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34861
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34861
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37825/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 01:06:57.547811 5749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:57.548187 5749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:57.548362 5749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:57.555776 5749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:57.555940 5749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:57.556035 5749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:57.556146 5749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:57.556232 5749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 01:06:57.556324 5749 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 01:06:57.556401 5749 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37825/jwks.json
I20260619 01:06:57.565728 5749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37825/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:57.567140 5749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:57.569015 5749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:57.580935 5754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:57.580868 5757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:57.580935 5755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:57.582366 5749 server_base.cc:1034] running on GCE node
I20260619 01:06:57.583294 5749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:57.584794 5749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:57.586035 5749 hybrid_clock.cc:648] HybridClock initialized: now 1781831217585986 us; error 46 us; skew 500 ppm
I20260619 01:06:57.586416 5749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:06:57.588850 5749 webserver.cc:460] Webserver started at http://127.0.97.62:46695/ using document root <none> and password file <none>
I20260619 01:06:57.589690 5749 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:57.589826 5749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:57.590184 5749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:57.593262 5749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "252d516c9ab647d39a22169c538f296e"
format_stamp: "Formatted at 2026-06-19 01:06:57 on dist-test-slave-5p8w"
server_key: "26ab0278a1a0e8c4eef645798c522d1e"
server_key_iv: "d6a012b6de91e9c68164e4e022a40aee"
server_key_version: "encryptionkey@0"
I20260619 01:06:57.594241 5749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "252d516c9ab647d39a22169c538f296e"
format_stamp: "Formatted at 2026-06-19 01:06:57 on dist-test-slave-5p8w"
server_key: "26ab0278a1a0e8c4eef645798c522d1e"
server_key_iv: "d6a012b6de91e9c68164e4e022a40aee"
server_key_version: "encryptionkey@0"
I20260619 01:06:57.600080 5749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:06:57.604033 5763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:57.605654 5749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 01:06:57.605891 5749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "252d516c9ab647d39a22169c538f296e"
format_stamp: "Formatted at 2026-06-19 01:06:57 on dist-test-slave-5p8w"
server_key: "26ab0278a1a0e8c4eef645798c522d1e"
server_key_iv: "d6a012b6de91e9c68164e4e022a40aee"
server_key_version: "encryptionkey@0"
I20260619 01:06:57.606139 5749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:57.621668 5749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:57.622581 5749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:57.622892 5749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:57.642282 5749 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34861
I20260619 01:06:57.642314 5814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34861 every 8 connection(s)
I20260619 01:06:57.643778 5749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:57.647779 5815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:57.650068 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 5749
I20260619 01:06:57.650275 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:57.650940 388 external_mini_cluster.cc:1428] Setting key 0c8128528b8ac2eec4dc6f53a6780734
I20260619 01:06:57.660351 5815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e: Bootstrap starting.
I20260619 01:06:57.663832 5815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:57.665072 5815 log.cc:824] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:57.665673 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:57.653037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.0.97.62:34861)
0619 01:06:57.654173 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:57.654186 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:57.654232 (+ 46us) server_negotiation.cc:407] Connection header received
0619 01:06:57.656071 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:57.656100 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:57.656772 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:57.657267 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:57.657951 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:57.659308 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:57.660801 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:57.661623 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:57.661857 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:57.663092 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:57.663106 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:57.663123 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:57.663699 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:57.663821 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:57.663843 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:57.663991 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:57.664200 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 01:06:57.664550 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":377,"threads_started":1}
I20260619 01:06:57.668501 5815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e: No bootstrap required, opened a new log
I20260619 01:06:57.670023 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:06:57.677672 5815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d516c9ab647d39a22169c538f296e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34861 } }
I20260619 01:06:57.678198 5815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:57.678359 5815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252d516c9ab647d39a22169c538f296e, State: Initialized, Role: FOLLOWER
I20260619 01:06:57.679138 5815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d516c9ab647d39a22169c538f296e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34861 } }
I20260619 01:06:57.679414 5815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:57.679574 5815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:57.679757 5815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:57.682191 5815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d516c9ab647d39a22169c538f296e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34861 } }
I20260619 01:06:57.682777 5815 leader_election.cc:302] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 252d516c9ab647d39a22169c538f296e; no voters:
I20260619 01:06:57.683430 5815 leader_election.cc:288] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:57.683625 5820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:57.684895 5820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [term 1 LEADER]: Becoming Leader. State: Replica: 252d516c9ab647d39a22169c538f296e, State: Running, Role: LEADER
I20260619 01:06:57.685510 5820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d516c9ab647d39a22169c538f296e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34861 } }
I20260619 01:06:57.685902 5815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:57.688743 5821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "252d516c9ab647d39a22169c538f296e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d516c9ab647d39a22169c538f296e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34861 } } }
I20260619 01:06:57.689126 5821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:57.689584 5822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252d516c9ab647d39a22169c538f296e. Latest consensus state: current_term: 1 leader_uuid: "252d516c9ab647d39a22169c538f296e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252d516c9ab647d39a22169c538f296e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34861 } } }
I20260619 01:06:57.689952 5822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:57.693111 5829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:57.698922 5829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:57.708731 5829 catalog_manager.cc:1347] Generated new cluster ID: 052e9376fc7046369c933b90933ce2c5
I20260619 01:06:57.708879 5829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:57.722779 5829 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:57.724763 5829 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:57.733757 5829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 252d516c9ab647d39a22169c538f296e: Generated new TSK 0
I20260619 01:06:57.734977 5829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:06:57.768533 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:57.760134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.0.97.62:34861)
0619 01:06:57.760410 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:57.760424 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:57.761124 (+ 700us) server_negotiation.cc:407] Connection header received
0619 01:06:57.761450 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:57.761454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:57.761539 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:57.761721 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0619 01:06:57.762241 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:57.763012 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:57.764509 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:57.764979 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:57.765156 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:57.765778 (+ 622us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 01:06:57.767796 (+ 2018us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 01:06:57.768166 (+ 370us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35162: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 01:06:57.768689 5818 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35162: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 01:06:57.769074 5844 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:06:57.759981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34861 (local address 127.0.0.1:35162)
0619 01:06:57.760984 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:57.761003 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:57.761210 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:57.761734 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:57.761739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:57.761751 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 01:06:57.762068 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:57.762078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:57.763184 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:57.763191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:57.764337 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:57.764351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:57.765126 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:57.765213 (+ 87us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:57.765448 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:57.765489 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 01:06:57.768031 (+ 2542us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 01:06:57.768301 (+ 270us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34861: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37825/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":674,"threads_started":1}
I20260619 01:06:57.770328 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 5749
I20260619 01:06:57.771958 5810 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 01:06:57.781111 5845 webserver.cc:533] Webserver: sslize failed:
E20260619 01:06:57.781227 5845 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 01:06:57.864166 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 5749
2026-06-19T01:06:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (743 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T01:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:06:57Z Disabled control of system clock
I20260619 01:06:57.948771 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44497
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44497
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/testchainca.cert with env {}
W20260619 01:06:58.288718 5850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:06:58.289042 5850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:06:58.289134 5850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:06:58.296638 5850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:06:58.296772 5850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:06:58.296808 5850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:06:58.296857 5850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:06:58.296901 5850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 01:06:58.296962 5850 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 01:06:58.297001 5850 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 01:06:58.307395 5850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:06:58.308768 5850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:06:58.310225 5850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:06:58.322830 5855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:58.323031 5856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:06:58.322853 5858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:06:58.323915 5850 server_base.cc:1034] running on GCE node
I20260619 01:06:58.325040 5850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:06:58.326843 5850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:06:58.328095 5850 hybrid_clock.cc:648] HybridClock initialized: now 1781831218328040 us; error 69 us; skew 500 ppm
I20260619 01:06:58.328522 5850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:06:58.335304 5850 webserver.cc:460] Webserver started at http://127.0.97.62:35459/ using document root <none> and password file <none>
I20260619 01:06:58.336181 5850 fs_manager.cc:359] Metadata directory not provided
I20260619 01:06:58.336349 5850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:06:58.336730 5850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:06:58.340240 5850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "80294108e66744d49314cc8a8c2fe68b"
format_stamp: "Formatted at 2026-06-19 01:06:58 on dist-test-slave-5p8w"
server_key: "55cb70750cb163ff117d050bcb198c5f"
server_key_iv: "1754508ce0df1a3c8caf5be84a850640"
server_key_version: "encryptionkey@0"
I20260619 01:06:58.341220 5850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "80294108e66744d49314cc8a8c2fe68b"
format_stamp: "Formatted at 2026-06-19 01:06:58 on dist-test-slave-5p8w"
server_key: "55cb70750cb163ff117d050bcb198c5f"
server_key_iv: "1754508ce0df1a3c8caf5be84a850640"
server_key_version: "encryptionkey@0"
I20260619 01:06:58.348286 5850 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 01:06:58.352669 5864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:06:58.354388 5850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:06:58.354566 5850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "80294108e66744d49314cc8a8c2fe68b"
format_stamp: "Formatted at 2026-06-19 01:06:58 on dist-test-slave-5p8w"
server_key: "55cb70750cb163ff117d050bcb198c5f"
server_key_iv: "1754508ce0df1a3c8caf5be84a850640"
server_key_version: "encryptionkey@0"
I20260619 01:06:58.354805 5850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:06:58.380088 5850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:06:58.381069 5850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:06:58.381413 5850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:06:58.401778 5850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44497
I20260619 01:06:58.401813 5915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44497 every 8 connection(s)
I20260619 01:06:58.403292 5850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:06:58.407230 5916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:06:58.411368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 5850
I20260619 01:06:58.411657 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:06:58.412380 388 external_mini_cluster.cc:1428] Setting key 7fe15a5f269b49d53b572f21e133a675
I20260619 01:06:58.418606 5916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b: Bootstrap starting.
I20260619 01:06:58.422463 5916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b: Neither blocks nor log segments found. Creating new log.
I20260619 01:06:58.423697 5916 log.cc:824] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:06:58.427407 5916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b: No bootstrap required, opened a new log
I20260619 01:06:58.427786 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:58.414537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.0.97.62:44497)
0619 01:06:58.415719 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:58.415732 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:58.415790 (+ 58us) server_negotiation.cc:407] Connection header received
0619 01:06:58.417710 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:58.417736 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:58.418275 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:58.418658 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:06:58.419343 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:58.420812 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:58.422784 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:58.423556 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:58.423718 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:58.425230 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:06:58.425241 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:06:58.425266 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:06:58.425903 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:06:58.425981 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:06:58.426005 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:06:58.426100 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 01:06:58.426350 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 01:06:58.426725 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":448,"threads_started":1}
I20260619 01:06:58.432123 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:06:58.436050 5916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80294108e66744d49314cc8a8c2fe68b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44497 } }
I20260619 01:06:58.436478 5916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:06:58.436601 5916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80294108e66744d49314cc8a8c2fe68b, State: Initialized, Role: FOLLOWER
I20260619 01:06:58.437214 5916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80294108e66744d49314cc8a8c2fe68b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44497 } }
I20260619 01:06:58.437554 5916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:06:58.437708 5916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:06:58.437896 5916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:06:58.440179 5916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80294108e66744d49314cc8a8c2fe68b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44497 } }
I20260619 01:06:58.440714 5916 leader_election.cc:302] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80294108e66744d49314cc8a8c2fe68b; no voters:
I20260619 01:06:58.441403 5916 leader_election.cc:288] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:06:58.441556 5921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:06:58.442773 5921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [term 1 LEADER]: Becoming Leader. State: Replica: 80294108e66744d49314cc8a8c2fe68b, State: Running, Role: LEADER
I20260619 01:06:58.443356 5921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80294108e66744d49314cc8a8c2fe68b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44497 } }
I20260619 01:06:58.443779 5916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:06:58.446143 5923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80294108e66744d49314cc8a8c2fe68b. Latest consensus state: current_term: 1 leader_uuid: "80294108e66744d49314cc8a8c2fe68b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80294108e66744d49314cc8a8c2fe68b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44497 } } }
I20260619 01:06:58.446714 5923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:58.446648 5922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80294108e66744d49314cc8a8c2fe68b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80294108e66744d49314cc8a8c2fe68b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44497 } } }
I20260619 01:06:58.447043 5922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b [sys.catalog]: This master's current role is: LEADER
I20260619 01:06:58.448581 5930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:06:58.453389 5930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:06:58.462664 5930 catalog_manager.cc:1347] Generated new cluster ID: e5a5f80cdc514ce9b2435f583d6cd4ca
I20260619 01:06:58.462821 5930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:06:58.484270 5930 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:06:58.485757 5930 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:06:58.492398 5930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80294108e66744d49314cc8a8c2fe68b: Generated new TSK 0
I20260619 01:06:58.493393 5930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:06:58.571863 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:06:58.515497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.0.97.62:44497)
0619 01:06:58.515764 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 01:06:58.515774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:06:58.517226 (+ 1452us) server_negotiation.cc:407] Connection header received
0619 01:06:58.517506 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:06:58.517511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:06:58.517593 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:06:58.517733 (+ 140us) server_negotiation.cc:226] Negotiated authn=JWT
0619 01:06:58.518333 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:58.519066 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:58.521255 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:06:58.521766 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:06:58.521882 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:58.522462 (+ 580us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 01:06:58.571236 (+ 48774us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 01:06:58.571591 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54426: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":96}
W20260619 01:06:58.572049 5919 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54426: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 01:06:58.572322 5945 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:06:58.515344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44497 (local address 127.0.0.1:54426)
0619 01:06:58.517086 (+ 1742us) negotiation.cc:107] Waiting for socket to connect
0619 01:06:58.517108 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:06:58.517316 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:06:58.517797 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:06:58.517803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:06:58.517816 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 01:06:58.518164 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:58.518176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:58.519230 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:58.519237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:58.521101 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:06:58.521119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:06:58.521904 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:06:58.521965 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:06:58.522151 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:06:58.522201 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 01:06:58.571563 (+ 49362us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 01:06:58.571896 (+ 333us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44497: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":1574,"thread_start_us":1413,"threads_started":1}
I20260619 01:06:58.573621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 5850
I20260619 01:06:58.575107 5911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:06:58.674422 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 5850
2026-06-19T01:06:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (789 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (435 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (433 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:06:59 dist-test-slave-5p8w krb5kdc[5960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:06:59 dist-test-slave-5p8w krb5kdc[5960](info): set up 2 sockets
Jun 19 01:06:59 dist-test-slave-5p8w krb5kdc[5960](info): commencing operation
krb5kdc: starting...
W20260619 01:07:00.654222 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.009s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:07:00 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:07:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:07:00.881930 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34241
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:01.234367 5973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:01.234695 5973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:01.234843 5973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:01.242520 5973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:07:01.242650 5973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:01.242725 5973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:01.242817 5973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:07:01.242902 5973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:07:01.253336 5973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:01.254784 5973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:01.256201 5973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:01.269593 5978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:01.269658 5979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:01.269635 5981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:01.270540 5973 server_base.cc:1034] running on GCE node
I20260619 01:07:01.271521 5973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:01.273167 5973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:01.274425 5973 hybrid_clock.cc:648] HybridClock initialized: now 1781831221274370 us; error 50 us; skew 500 ppm
I20260619 01:07:01.274868 5973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:01 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:01.278375 5973 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:07:01.280143 5973 webserver.cc:460] Webserver started at http://127.0.97.62:39067/ using document root <none> and password file <none>
I20260619 01:07:01.280912 5973 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:01.281025 5973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:01.281383 5973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:01.284623 5973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "3a65574d116d4d3e87517233451d0872"
format_stamp: "Formatted at 2026-06-19 01:07:01 on dist-test-slave-5p8w"
server_key: "483ab822bf2415f54ea2b9ab2e8e8607"
server_key_iv: "f2fa4aca82ca97106364dc663216dbb8"
server_key_version: "encryptionkey@0"
I20260619 01:07:01.285527 5973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "3a65574d116d4d3e87517233451d0872"
format_stamp: "Formatted at 2026-06-19 01:07:01 on dist-test-slave-5p8w"
server_key: "483ab822bf2415f54ea2b9ab2e8e8607"
server_key_iv: "f2fa4aca82ca97106364dc663216dbb8"
server_key_version: "encryptionkey@0"
I20260619 01:07:01.291457 5973 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 01:07:01.295432 5988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:01.297210 5973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:07:01.297467 5973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "3a65574d116d4d3e87517233451d0872"
format_stamp: "Formatted at 2026-06-19 01:07:01 on dist-test-slave-5p8w"
server_key: "483ab822bf2415f54ea2b9ab2e8e8607"
server_key_iv: "f2fa4aca82ca97106364dc663216dbb8"
server_key_version: "encryptionkey@0"
I20260619 01:07:01.297742 5973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:01.312841 5973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:01.318212 5973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:01.318537 5973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:01.338161 5973 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34241
I20260619 01:07:01.338202 6039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34241 every 8 connection(s)
I20260619 01:07:01.339695 5973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:07:01.343802 6040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:01.344269 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 5973
I20260619 01:07:01.344502 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:07:01.345116 388 external_mini_cluster.cc:1428] Setting key 62109208950e3fdf6488938104a4ac2d
I20260619 01:07:01.359797 6040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872: Bootstrap starting.
Jun 19 01:07:01 dist-test-slave-5p8w krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:01.363394 6040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:01.364562 6040 log.cc:824] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:01.367946 6040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872: No bootstrap required, opened a new log
I20260619 01:07:01.372054 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:01.347436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59104 (local address 127.0.97.62:34241)
0619 01:07:01.348312 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:01.348326 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:01.348374 (+ 48us) server_negotiation.cc:407] Connection header received
0619 01:07:01.350279 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:01.350309 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:01.350810 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:01.351200 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:01.353245 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:01.354705 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:01.356607 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:01.358099 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:01.359150 (+ 1051us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:01.363506 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:01.363516 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:01.363562 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:01.363632 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:01.367165 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:01.368019 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:01.368036 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:01.368044 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:01.368288 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:01.368732 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:01.368740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:01.368744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:01.369475 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:01.369825 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:01.370323 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 01:07:01.370847 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":392,"threads_started":1}
I20260619 01:07:01.377586 6040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a65574d116d4d3e87517233451d0872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34241 } }
I20260619 01:07:01.378036 6040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:01.378149 6040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a65574d116d4d3e87517233451d0872, State: Initialized, Role: FOLLOWER
I20260619 01:07:01.378832 6040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a65574d116d4d3e87517233451d0872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34241 } }
I20260619 01:07:01.379091 6040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:07:01.379240 6040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:07:01.379444 6040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:01.381827 6040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a65574d116d4d3e87517233451d0872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34241 } }
I20260619 01:07:01.382409 6040 leader_election.cc:302] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a65574d116d4d3e87517233451d0872; no voters:
I20260619 01:07:01.383112 6040 leader_election.cc:288] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:07:01.383231 6045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:01.383967 6045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [term 1 LEADER]: Becoming Leader. State: Replica: 3a65574d116d4d3e87517233451d0872, State: Running, Role: LEADER
I20260619 01:07:01.384621 6045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a65574d116d4d3e87517233451d0872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34241 } }
I20260619 01:07:01.385630 6040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:07:01.388250 6047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a65574d116d4d3e87517233451d0872. Latest consensus state: current_term: 1 leader_uuid: "3a65574d116d4d3e87517233451d0872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a65574d116d4d3e87517233451d0872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34241 } } }
I20260619 01:07:01.388746 6047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:01.393054 6046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a65574d116d4d3e87517233451d0872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a65574d116d4d3e87517233451d0872" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34241 } } }
I20260619 01:07:01.393426 6046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:01.394528 6053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:07:01.399992 6053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:07:01.409955 6053 catalog_manager.cc:1347] Generated new cluster ID: b0c256fef13c4f52b17ef943431cca76
I20260619 01:07:01.410108 6053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:07:01.422637 6053 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:07:01.424108 6053 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:07:01.440326 6053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a65574d116d4d3e87517233451d0872: Generated new TSK 0
I20260619 01:07:01.441612 6053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:07:01.505581 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34241
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:01.866133 6066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:01.866568 6066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:01.866750 6066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:01.874577 6066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:01.874712 6066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:01.874891 6066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:07:01.885435 6066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:01.886981 6066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:01.888484 6066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:01.903111 6071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:01.903160 6072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:01.903134 6074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:01.904073 6066 server_base.cc:1034] running on GCE node
I20260619 01:07:01.904878 6066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:01.906085 6066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:01.907299 6066 hybrid_clock.cc:648] HybridClock initialized: now 1781831221907219 us; error 68 us; skew 500 ppm
I20260619 01:07:01.907666 6066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:01 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:01.911322 6066 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:07:01.913054 6066 webserver.cc:460] Webserver started at http://127.0.97.1:45445/ using document root <none> and password file <none>
I20260619 01:07:01.913942 6066 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:01.914122 6066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:01.914459 6066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:01.917838 6066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "c99ad45e98804dd4bbb1504a9762a3f2"
format_stamp: "Formatted at 2026-06-19 01:07:01 on dist-test-slave-5p8w"
server_key: "b410641dc95de649d1e341cfccc06018"
server_key_iv: "f6c3f15ba5c3dc6f4942f1c6de9f9787"
server_key_version: "encryptionkey@0"
I20260619 01:07:01.918766 6066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c99ad45e98804dd4bbb1504a9762a3f2"
format_stamp: "Formatted at 2026-06-19 01:07:01 on dist-test-slave-5p8w"
server_key: "b410641dc95de649d1e341cfccc06018"
server_key_iv: "f6c3f15ba5c3dc6f4942f1c6de9f9787"
server_key_version: "encryptionkey@0"
I20260619 01:07:01.924834 6066 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 01:07:01.929205 6081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:01.931027 6066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:07:01.931257 6066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "c99ad45e98804dd4bbb1504a9762a3f2"
format_stamp: "Formatted at 2026-06-19 01:07:01 on dist-test-slave-5p8w"
server_key: "b410641dc95de649d1e341cfccc06018"
server_key_iv: "f6c3f15ba5c3dc6f4942f1c6de9f9787"
server_key_version: "encryptionkey@0"
I20260619 01:07:01.931499 6066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:01.956568 6066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:01.961256 6066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:01.961644 6066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:01.962702 6066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:01.964182 6066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:01.964293 6066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:01.964444 6066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:01.964519 6066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:02.003166 6066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40635
I20260619 01:07:02.003226 6193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40635 every 8 connection(s)
I20260619 01:07:02.004596 6066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:07:02.011121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 6066
I20260619 01:07:02.011376 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:07:02.012199 388 external_mini_cluster.cc:1428] Setting key 9e3a4e37e377cc63fbc96be5e6ea4a32
Jun 19 01:07:02 dist-test-slave-5p8w krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831221, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:02.027808 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:02.007645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57959 (local address 127.0.97.62:34241)
0619 01:07:02.008084 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:02.008088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:02.009256 (+ 1168us) server_negotiation.cc:407] Connection header received
0619 01:07:02.011745 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:02.011752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:02.011867 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:02.011989 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:02.014113 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:02.014829 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:02.016558 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:02.016969 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:02.017125 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:02.021440 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:02.021446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:02.021450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:02.021490 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:02.024185 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:02.025121 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:02.025126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:02.025128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:02.025253 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:02.025827 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:02.025834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:02.025842 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:02.026122 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:02.026287 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:02.027167 (+ 880us) server_negotiation.cc:299] Negotiation successful
0619 01:07:02.027419 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":222,"threads_started":1}
I20260619 01:07:02.028916 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:02.008305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34241 (local address 127.0.97.1:57959)
0619 01:07:02.009065 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:02.009121 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:02.011198 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:02.012317 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:02.012340 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:02.013002 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:02.013802 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:02.013823 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:02.015018 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:02.015022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:02.016386 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:02.016400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:02.017222 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:02.017231 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:02.017560 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:02.018342 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:02.018368 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:02.021075 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:02.024399 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:02.024411 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:02.024438 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:02.024946 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:02.025454 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:02.025460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:02.025464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:02.025693 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:02.026270 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:02.026278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:02.026681 (+ 403us) client_negotiation.cc:769] Sending connection context
0619 01:07:02.027033 (+ 352us) client_negotiation.cc:240] Negotiation successful
0619 01:07:02.027684 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":306,"threads_started":1}
I20260619 01:07:02.030856 6194 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34241
I20260619 01:07:02.031255 6194 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:07:02.032325 6194 heartbeater.cc:511] Master 127.0.97.62:34241 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:07:02.035125 6005 ts_manager.cc:194] Registered new tserver with Master: c99ad45e98804dd4bbb1504a9762a3f2 (127.0.97.1:40635)
I20260619 01:07:02.036954 6005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:07:02.062158 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34241
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:02.407021 6200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:02.407394 6200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:02.407580 6200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:02.415472 6200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:02.415704 6200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:02.415913 6200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:07:02.426970 6200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:02.428472 6200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:02.430022 6200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:02.444722 6205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:02.444788 6208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:02.444810 6206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:02.446015 6200 server_base.cc:1034] running on GCE node
I20260619 01:07:02.446759 6200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:02.447842 6200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:02.449056 6200 hybrid_clock.cc:648] HybridClock initialized: now 1781831222449004 us; error 57 us; skew 500 ppm
I20260619 01:07:02.449468 6200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:02 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:02.453011 6200 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:07:02.454794 6200 webserver.cc:460] Webserver started at http://127.0.97.2:36875/ using document root <none> and password file <none>
I20260619 01:07:02.455513 6200 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:02.455624 6200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:02.455888 6200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:02.459071 6200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "485f00d952ac4dffaa397fc8bb752df0"
format_stamp: "Formatted at 2026-06-19 01:07:02 on dist-test-slave-5p8w"
server_key: "62db60b77b8f0c92755943a834c49b8b"
server_key_iv: "355b33e7ff6c1295277e1f3db69b403e"
server_key_version: "encryptionkey@0"
I20260619 01:07:02.459882 6200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "485f00d952ac4dffaa397fc8bb752df0"
format_stamp: "Formatted at 2026-06-19 01:07:02 on dist-test-slave-5p8w"
server_key: "62db60b77b8f0c92755943a834c49b8b"
server_key_iv: "355b33e7ff6c1295277e1f3db69b403e"
server_key_version: "encryptionkey@0"
I20260619 01:07:02.465961 6200 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 01:07:02.470127 6215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:02.471838 6200 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:07:02.472077 6200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "485f00d952ac4dffaa397fc8bb752df0"
format_stamp: "Formatted at 2026-06-19 01:07:02 on dist-test-slave-5p8w"
server_key: "62db60b77b8f0c92755943a834c49b8b"
server_key_iv: "355b33e7ff6c1295277e1f3db69b403e"
server_key_version: "encryptionkey@0"
I20260619 01:07:02.472347 6200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:02.496359 6200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:02.501215 6200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:02.501608 6200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:02.502761 6200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:02.504104 6200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:02.504182 6200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:02.504266 6200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:02.504310 6200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:02.538682 6200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35445
I20260619 01:07:02.538775 6327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35445 every 8 connection(s)
I20260619 01:07:02.540025 6200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:07:02.544580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 6200
I20260619 01:07:02.544797 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:07:02.545212 388 external_mini_cluster.cc:1428] Setting key 48f14a9d51a526b85f7369821eeeb1a1
Jun 19 01:07:02 dist-test-slave-5p8w krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:02.564978 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:02.542816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57895 (local address 127.0.97.62:34241)
0619 01:07:02.543268 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:02.543273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:02.544341 (+ 1068us) server_negotiation.cc:407] Connection header received
0619 01:07:02.546542 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:02.546548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:02.546643 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:02.546764 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:02.548998 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:02.549797 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:02.552392 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:02.552832 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:02.552928 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:02.557065 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:02.557075 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:02.557083 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:02.557131 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:02.560881 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:02.561897 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:02.561906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:02.561912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:02.562089 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:02.562704 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:02.562712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:02.562718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:02.563117 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:02.563276 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:02.564187 (+ 911us) server_negotiation.cc:299] Negotiation successful
0619 01:07:02.564451 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":230,"threads_started":1}
I20260619 01:07:02.565730 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:02.543401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34241 (local address 127.0.97.2:57895)
0619 01:07:02.544173 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:02.544213 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:02.546254 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:02.547041 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:02.547051 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:02.547845 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:02.548788 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:02.548818 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:02.549957 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:02.549965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:02.552220 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:02.552240 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:02.553092 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:02.553101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:02.553481 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:02.554249 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:02.554274 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:02.556784 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:02.561104 (+ 4320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:02.561112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:02.561129 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:02.561752 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:02.562278 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:02.562282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:02.562285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:02.562545 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:02.563272 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:02.563279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:02.563655 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 01:07:02.564033 (+ 378us) client_negotiation.cc:240] Negotiation successful
0619 01:07:02.564485 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":301,"threads_started":1}
I20260619 01:07:02.567458 6328 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34241
I20260619 01:07:02.567808 6328 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:07:02.568794 6328 heartbeater.cc:511] Master 127.0.97.62:34241 requested a full tablet report, sending...
I20260619 01:07:02.570451 6005 ts_manager.cc:194] Registered new tserver with Master: 485f00d952ac4dffaa397fc8bb752df0 (127.0.97.2:35445)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 01:07:02.571542 6005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:07:02.596830 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34241
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:02.943286 6334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:02.943612 6334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:02.943783 6334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:02.951784 6334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:02.951926 6334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:02.952114 6334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:07:02.962706 6334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:02.964146 6334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:02.965566 6334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:02.980337 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:02.980247 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:02.980399 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:02.981153 6334 server_base.cc:1034] running on GCE node
I20260619 01:07:02.981870 6334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:02.982910 6334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:02.984157 6334 hybrid_clock.cc:648] HybridClock initialized: now 1781831222984106 us; error 45 us; skew 500 ppm
I20260619 01:07:02.984503 6334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:02 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:02.988060 6334 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:07:02.989638 6334 webserver.cc:460] Webserver started at http://127.0.97.3:37341/ using document root <none> and password file <none>
I20260619 01:07:02.990461 6334 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:02.990584 6334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:02.990899 6334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:02.994036 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "3190f0694b9b4f76a44263727264d385"
format_stamp: "Formatted at 2026-06-19 01:07:02 on dist-test-slave-5p8w"
server_key: "6459039799bd367ee7d2560baad89192"
server_key_iv: "98f2efdbdd7787a6b736a18b3a6940ec"
server_key_version: "encryptionkey@0"
I20260619 01:07:02.994812 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3190f0694b9b4f76a44263727264d385"
format_stamp: "Formatted at 2026-06-19 01:07:02 on dist-test-slave-5p8w"
server_key: "6459039799bd367ee7d2560baad89192"
server_key_iv: "98f2efdbdd7787a6b736a18b3a6940ec"
server_key_version: "encryptionkey@0"
I20260619 01:07:03.000384 6334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:07:03.004788 6349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:03.006515 6334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:03.006726 6334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "3190f0694b9b4f76a44263727264d385"
format_stamp: "Formatted at 2026-06-19 01:07:02 on dist-test-slave-5p8w"
server_key: "6459039799bd367ee7d2560baad89192"
server_key_iv: "98f2efdbdd7787a6b736a18b3a6940ec"
server_key_version: "encryptionkey@0"
I20260619 01:07:03.006954 6334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:03.032272 6334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:03.037133 6334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:03.037490 6334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:03.038547 6334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:03.039981 6334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:03.040093 6334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:03.040251 6334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:03.040325 6334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:03.040776 6194 heartbeater.cc:503] Master 127.0.97.62:34241 was elected leader, sending a full tablet report...
I20260619 01:07:03.074803 6334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33675
I20260619 01:07:03.074834 6461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33675 every 8 connection(s)
I20260619 01:07:03.076201 6334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:07:03.079195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 6334
I20260619 01:07:03.079422 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:07:03.079916 388 external_mini_cluster.cc:1428] Setting key 4e7329bdb3971c54cdf87c2180f2bbb8
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831222, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:03.097880 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:03.079046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38897 (local address 127.0.97.62:34241)
0619 01:07:03.079622 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:03.079629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:03.080778 (+ 1149us) server_negotiation.cc:407] Connection header received
0619 01:07:03.082762 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:03.082767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:03.082856 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:03.082986 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:03.084559 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.085447 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.087243 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.087648 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.087782 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:03.091817 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:03.091823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:03.091827 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:03.091864 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:03.094894 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.095732 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.095737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.095739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.095854 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.096267 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.096271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.096274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.096554 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:03.096693 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:03.097430 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 01:07:03.097600 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":276,"threads_started":1}
I20260619 01:07:03.098883 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:03.079671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34241 (local address 127.0.97.3:38897)
0619 01:07:03.080592 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:03.080643 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:03.082482 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:03.083241 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:03.083250 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:03.083783 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:03.084376 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:03.084393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.085616 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.085620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:03.087085 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:03.087098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.087893 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.087900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:03.088132 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:03.088892 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:03.088915 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:03.091578 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:03.095045 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:03.095053 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:03.095069 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:03.095622 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:03.095946 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:03.095949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:03.095951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:03.096133 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:03.096666 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:03.096674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:03.097016 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 01:07:03.097265 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 01:07:03.097749 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":420,"threads_started":1}
I20260619 01:07:03.100381 6462 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34241
I20260619 01:07:03.100654 6462 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:03.101469 6462 heartbeater.cc:511] Master 127.0.97.62:34241 requested a full tablet report, sending...
I20260619 01:07:03.102952 6005 ts_manager.cc:194] Registered new tserver with Master: 3190f0694b9b4f76a44263727264d385 (127.0.97.3:33675)
I20260619 01:07:03.103744 6005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38897
I20260619 01:07:03.107803 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/test-admin.keytab.
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:03.136621 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 01:07:03.142243 6470 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:03.139676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.0.97.1:40635)
0619 01:07:03.140334 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:03.140341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:03.140371 (+ 30us) server_negotiation.cc:407] Connection header received
0619 01:07:03.140448 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:03.140455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:03.140678 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:03.140808 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:03.141767 (+ 959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56700: BlockingRecv error: recv got EOF from 127.0.0.1:56700 (error 108)
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":330,"threads_started":1}
W20260619 01:07:03.142395 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:03.144351 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:03.145848 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:07:03.162704 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:03.147620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.97.1:40635)
0619 01:07:03.147861 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:03.147867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:03.148119 (+ 252us) server_negotiation.cc:407] Connection header received
0619 01:07:03.148336 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:03.148342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:03.148419 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:03.148541 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:03.149598 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.150606 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.151869 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.152470 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.152613 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:03.156386 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:03.156401 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:03.156418 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:03.156457 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:03.159590 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.160426 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.160436 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.160439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.160570 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.160966 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.160972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.160975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.161433 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:03.161755 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:03.162120 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 01:07:03.162395 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 01:07:03.168920 6470 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:03.167014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.0.97.1:40635)
0619 01:07:03.167204 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:03.167208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:03.167598 (+ 390us) server_negotiation.cc:407] Connection header received
0619 01:07:03.167737 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:03.167741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:03.167802 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:03.167885 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:03.168621 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56730: BlockingRecv error: recv got EOF from 127.0.0.1:56730 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 01:07:03.169516 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5cc)
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:03.171252 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:03.172637 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:07:03.187986 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:03.174294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.0.97.1:40635)
0619 01:07:03.174541 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:03.174548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:03.174750 (+ 202us) server_negotiation.cc:407] Connection header received
0619 01:07:03.174889 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:03.174893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:03.174956 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:03.175042 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:03.176041 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.176784 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.178202 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.178638 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.178789 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:03.182668 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:03.182674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:03.182678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:03.182728 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:03.185144 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.186012 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.186018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.186022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.186200 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.186612 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.186617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.186621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.186912 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:03.187189 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:03.187473 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 01:07:03.187691 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 01:07:03.192775 6470 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:03.190788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.0.97.1:40635)
0619 01:07:03.190979 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:03.190988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:03.191321 (+ 333us) server_negotiation.cc:407] Connection header received
0619 01:07:03.191559 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:03.191564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:03.191633 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:03.191744 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:03.192515 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56760: BlockingRecv error: recv got EOF from 127.0.0.1:56760 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 01:07:03.192940 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781831171286417-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:03.194536 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:03.195860 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:07:03.210341 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:03.197418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.0.97.1:40635)
0619 01:07:03.197635 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:03.197641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:03.197785 (+ 144us) server_negotiation.cc:407] Connection header received
0619 01:07:03.197912 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:03.197916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:03.197980 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:03.198064 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:03.199032 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.199744 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.200974 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.201462 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.201565 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:03.205239 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:03.205245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:03.205249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:03.205346 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:03.207764 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.208503 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.208506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.208509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.208659 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.208985 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.208988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.208990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.209276 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:03.209590 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:03.209788 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 01:07:03.210044 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:07:03.211958 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 6066
I20260619 01:07:03.224642 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:03.212783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.0.97.1:40635)
0619 01:07:03.212966 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:03.212973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:03.213259 (+ 286us) server_negotiation.cc:407] Connection header received
0619 01:07:03.213497 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:03.213501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:03.213559 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:03.213661 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:03.214693 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.215378 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.216736 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.217161 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.217327 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:03.219436 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:03.219441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:03.219444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:03.219481 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:03.221923 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.222659 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.222676 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.222684 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.222823 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.223210 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.223217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.223221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.223485 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:03.223764 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:03.224100 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 01:07:03.224327 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 01:07:03.225227 6189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:03.309926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 6066
I20260619 01:07:03.326347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 6200
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:07:03.346078 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:03.327272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.0.97.2:35445)
0619 01:07:03.327877 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:03.327886 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:03.327913 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:07:03.327971 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:03.327977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:03.328220 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:03.328356 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:03.329576 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.330591 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.331823 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.332400 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.332504 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:03.336594 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:03.336609 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:03.336625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:03.336675 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:03.342872 (+ 6197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.343865 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.343873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.343875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.344007 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.344394 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.344397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.344399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.344755 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:03.344978 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:03.345529 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 01:07:03.345767 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":306,"threads_started":1}
I20260619 01:07:03.347414 6323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:03.433218 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 6200
I20260619 01:07:03.449469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 6334
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:07:03.465274 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:03.450409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.0.97.3:33675)
0619 01:07:03.451046 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:03.451052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:03.451082 (+ 30us) server_negotiation.cc:407] Connection header received
0619 01:07:03.451141 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:03.451147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:03.451367 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:03.451545 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:03.452630 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.453702 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.454958 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:03.455529 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:03.455682 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:03.459670 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:03.459684 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:03.459700 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:03.459740 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:03.462445 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.463185 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.463193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.463196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.463320 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:03.463713 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:03.463719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:03.463723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:03.464074 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:03.464399 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:03.464704 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 01:07:03.464966 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":346,"threads_started":1}
I20260619 01:07:03.466735 6457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:03.550817 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 6334
I20260619 01:07:03.566912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 5973
I20260619 01:07:03.568130 6035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:03.670835 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 5973
2026-06-19T01:07:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4133 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[6490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[6490](info): set up 2 sockets
Jun 19 01:07:03 dist-test-slave-5p8w krb5kdc[6490](info): commencing operation
krb5kdc: starting...
W20260619 01:07:04.786142 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:07:04 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:07:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 01:07:05.004755 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40437
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:05.362345 6503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:05.362694 6503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:05.362872 6503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:05.370550 6503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:07:05.370699 6503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:05.370803 6503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:05.370929 6503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:07:05.371014 6503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:07:05.383106 6503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:05.384689 6503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:05.386248 6503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:05.399000 6508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:05.398952 6511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:05.399034 6509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:05.399797 6503 server_base.cc:1034] running on GCE node
I20260619 01:07:05.400658 6503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:05.402297 6503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:05.403548 6503 hybrid_clock.cc:648] HybridClock initialized: now 1781831225403498 us; error 51 us; skew 500 ppm
I20260619 01:07:05.403926 6503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:05 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831225, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:05.407256 6503 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 01:07:05.408813 6503 webserver.cc:460] Webserver started at http://127.0.97.62:44357/ using document root <none> and password file <none>
I20260619 01:07:05.409621 6503 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:05.409731 6503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:05.409989 6503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:05.413090 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "7cf28be23db947b1be7b7bc45f6e54a2"
format_stamp: "Formatted at 2026-06-19 01:07:05 on dist-test-slave-5p8w"
server_key: "b6c5d4c813e532042fa384585303050b"
server_key_iv: "c3e948db01634f9e9dcf21ccf6f13a2c"
server_key_version: "encryptionkey@0"
I20260619 01:07:05.413936 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "7cf28be23db947b1be7b7bc45f6e54a2"
format_stamp: "Formatted at 2026-06-19 01:07:05 on dist-test-slave-5p8w"
server_key: "b6c5d4c813e532042fa384585303050b"
server_key_iv: "c3e948db01634f9e9dcf21ccf6f13a2c"
server_key_version: "encryptionkey@0"
I20260619 01:07:05.419782 6503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:07:05.423708 6518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:05.425588 6503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:05.425818 6503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "7cf28be23db947b1be7b7bc45f6e54a2"
format_stamp: "Formatted at 2026-06-19 01:07:05 on dist-test-slave-5p8w"
server_key: "b6c5d4c813e532042fa384585303050b"
server_key_iv: "c3e948db01634f9e9dcf21ccf6f13a2c"
server_key_version: "encryptionkey@0"
I20260619 01:07:05.426049 6503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:05.453413 6503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:05.459525 6503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:05.459882 6503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:05.480010 6503 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40437
I20260619 01:07:05.480033 6569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40437 every 8 connection(s)
I20260619 01:07:05.481590 6503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:07:05.485572 6570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:05.486492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 6503
I20260619 01:07:05.486725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:07:05.487375 388 external_mini_cluster.cc:1428] Setting key 9ceffee239cf182e0589ae7279292f21
I20260619 01:07:05.500711 6570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2: Bootstrap starting.
Jun 19 01:07:05 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:05.504205 6570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:05.505739 6570 log.cc:824] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:05.508880 6570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2: No bootstrap required, opened a new log
I20260619 01:07:05.511833 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:05.489609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.0.97.62:40437)
0619 01:07:05.490451 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:05.490468 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:05.490521 (+ 53us) server_negotiation.cc:407] Connection header received
0619 01:07:05.492394 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:05.492429 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:05.493086 (+ 657us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:05.493630 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:05.495052 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:05.497106 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:05.498499 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:05.499596 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:05.499794 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:05.503809 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:05.503820 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:05.503842 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:05.503891 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:05.507489 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:05.508267 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:05.508274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:05.508281 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:05.508446 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:05.508832 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:05.508844 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:05.508846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:05.509594 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:05.509911 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:05.510335 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 01:07:05.510780 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":340,"threads_started":1}
I20260619 01:07:05.517444 6570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } }
I20260619 01:07:05.517856 6570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:05.517964 6570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf28be23db947b1be7b7bc45f6e54a2, State: Initialized, Role: FOLLOWER
I20260619 01:07:05.518533 6570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } }
I20260619 01:07:05.518782 6570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:07:05.518929 6570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:07:05.519100 6570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:05.521726 6570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } }
I20260619 01:07:05.522243 6570 leader_election.cc:302] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cf28be23db947b1be7b7bc45f6e54a2; no voters:
I20260619 01:07:05.522869 6570 leader_election.cc:288] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:07:05.523001 6575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:05.523687 6575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 1 LEADER]: Becoming Leader. State: Replica: 7cf28be23db947b1be7b7bc45f6e54a2, State: Running, Role: LEADER
I20260619 01:07:05.524247 6575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } }
I20260619 01:07:05.525064 6570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:07:05.526999 6576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } }
I20260619 01:07:05.527395 6576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:05.527805 6577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cf28be23db947b1be7b7bc45f6e54a2. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } }
I20260619 01:07:05.528134 6577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:05.530486 6582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:07:05.535745 6582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:07:05.545461 6582 catalog_manager.cc:1347] Generated new cluster ID: 6ef0b99fb5d24edda85c4a6401c5bfc3
I20260619 01:07:05.545627 6582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:07:05.555667 6582 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:07:05.557050 6582 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:07:05.567395 6582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2: Generated new TSK 0
I20260619 01:07:05.568301 6582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 01:07:05.616848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:05.952694 6596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:05.953008 6596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:05.953176 6596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:05.960852 6596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:05.960981 6596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:05.961176 6596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:07:05.971585 6596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:05.972998 6596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:05.974450 6596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:05.988615 6602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:05.988641 6601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:05.988590 6604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:05.989434 6596 server_base.cc:1034] running on GCE node
I20260619 01:07:05.990175 6596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:05.991215 6596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:05.992448 6596 hybrid_clock.cc:648] HybridClock initialized: now 1781831225992398 us; error 54 us; skew 500 ppm
I20260619 01:07:05.992795 6596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:05 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831225, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:05.996212 6596 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 01:07:05.997797 6596 webserver.cc:460] Webserver started at http://127.0.97.1:35289/ using document root <none> and password file <none>
I20260619 01:07:05.998533 6596 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:05.998679 6596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:05.998991 6596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:06.002072 6596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "bd4233d57aad4aa991a36d8abcd4913a"
format_stamp: "Formatted at 2026-06-19 01:07:05 on dist-test-slave-5p8w"
server_key: "5e33be9871e1972ea274b77382e61440"
server_key_iv: "ee167aaa84a81009adac73be3eb06855"
server_key_version: "encryptionkey@0"
I20260619 01:07:06.002892 6596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bd4233d57aad4aa991a36d8abcd4913a"
format_stamp: "Formatted at 2026-06-19 01:07:05 on dist-test-slave-5p8w"
server_key: "5e33be9871e1972ea274b77382e61440"
server_key_iv: "ee167aaa84a81009adac73be3eb06855"
server_key_version: "encryptionkey@0"
I20260619 01:07:06.008689 6596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:07:06.012667 6611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:06.014513 6596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:06.014716 6596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "bd4233d57aad4aa991a36d8abcd4913a"
format_stamp: "Formatted at 2026-06-19 01:07:05 on dist-test-slave-5p8w"
server_key: "5e33be9871e1972ea274b77382e61440"
server_key_iv: "ee167aaa84a81009adac73be3eb06855"
server_key_version: "encryptionkey@0"
I20260619 01:07:06.014941 6596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:06.037096 6596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:06.041918 6596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:06.042243 6596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:06.043277 6596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:06.044621 6596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:06.044726 6596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:06.044876 6596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:06.044948 6596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:06.079352 6596 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45717
I20260619 01:07:06.079396 6723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45717 every 8 connection(s)
I20260619 01:07:06.080732 6596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:07:06.088662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 6596
I20260619 01:07:06.088901 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:07:06.089610 388 external_mini_cluster.cc:1428] Setting key 741994b25bcbbd04885e9d59a8cc3e6a
Jun 19 01:07:06 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831225, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:06.102537 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:06.083445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32913 (local address 127.0.97.62:40437)
0619 01:07:06.083917 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:06.083921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:06.084841 (+ 920us) server_negotiation.cc:407] Connection header received
0619 01:07:06.086791 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:06.086797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:06.086916 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:06.087038 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:06.088729 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:06.089644 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:06.091561 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:06.092153 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:06.092293 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:06.096490 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:06.096496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:06.096500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:06.096540 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:06.099224 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:06.100081 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:06.100087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:06.100090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:06.100219 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:06.100675 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:06.100681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:06.100686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:06.100953 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:06.101115 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:06.101924 (+ 809us) server_negotiation.cc:299] Negotiation successful
0619 01:07:06.102162 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":262,"threads_started":1}
I20260619 01:07:06.103617 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:06.083944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.1:32913)
0619 01:07:06.084663 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:06.084713 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:06.086496 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:06.087309 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:06.087318 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:06.087888 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:06.088536 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:06.088555 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:06.089847 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:06.089851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:06.091376 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:06.091393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:06.092388 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:06.092392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:06.092616 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:06.093412 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:06.093442 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:06.096144 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:06.099420 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:06.099430 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:06.099457 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:06.099929 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:06.100339 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:06.100344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:06.100347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:06.100558 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:06.101084 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:06.101092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:06.101537 (+ 445us) client_negotiation.cc:769] Sending connection context
0619 01:07:06.101832 (+ 295us) client_negotiation.cc:240] Negotiation successful
0619 01:07:06.102404 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":267,"threads_started":1}
I20260619 01:07:06.105494 6724 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40437
I20260619 01:07:06.105863 6724 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:06.106638 6724 heartbeater.cc:511] Master 127.0.97.62:40437 requested a full tablet report, sending...
I20260619 01:07:06.108860 6535 ts_manager.cc:194] Registered new tserver with Master: bd4233d57aad4aa991a36d8abcd4913a (127.0.97.1:45717)
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:07:06.110570 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 01:07:06.139003 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:06.477104 6730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:06.477460 6730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:06.477684 6730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:06.488813 6730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:06.488967 6730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:06.489154 6730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:07:06.504472 6730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:06.505903 6730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:06.507236 6730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:06.521873 6736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:06.521840 6738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:06.521909 6735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:06.522662 6730 server_base.cc:1034] running on GCE node
I20260619 01:07:06.523353 6730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:06.524430 6730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:06.525676 6730 hybrid_clock.cc:648] HybridClock initialized: now 1781831226525622 us; error 51 us; skew 500 ppm
I20260619 01:07:06.526118 6730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:06 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:06.529726 6730 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 01:07:06.531446 6730 webserver.cc:460] Webserver started at http://127.0.97.2:41853/ using document root <none> and password file <none>
I20260619 01:07:06.532224 6730 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:06.532377 6730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:06.532704 6730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:06.535903 6730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "a6e5e05e6f2a4c08a8454513dbd696a7"
format_stamp: "Formatted at 2026-06-19 01:07:06 on dist-test-slave-5p8w"
server_key: "52d45b37fe3155ed64224da013cc7987"
server_key_iv: "2858254bebec73bc82795a9df140a719"
server_key_version: "encryptionkey@0"
I20260619 01:07:06.536789 6730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a6e5e05e6f2a4c08a8454513dbd696a7"
format_stamp: "Formatted at 2026-06-19 01:07:06 on dist-test-slave-5p8w"
server_key: "52d45b37fe3155ed64224da013cc7987"
server_key_iv: "2858254bebec73bc82795a9df140a719"
server_key_version: "encryptionkey@0"
I20260619 01:07:06.542620 6730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260619 01:07:06.546556 6745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:06.548038 6730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:07:06.548264 6730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "a6e5e05e6f2a4c08a8454513dbd696a7"
format_stamp: "Formatted at 2026-06-19 01:07:06 on dist-test-slave-5p8w"
server_key: "52d45b37fe3155ed64224da013cc7987"
server_key_iv: "2858254bebec73bc82795a9df140a719"
server_key_version: "encryptionkey@0"
I20260619 01:07:06.548518 6730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:06.569610 6730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:06.574517 6730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:06.574851 6730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:06.575947 6730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:06.577486 6730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:06.577601 6730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:06.577869 6730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:06.577970 6730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:06.613412 6730 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42509
I20260619 01:07:06.613461 6857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42509 every 8 connection(s)
I20260619 01:07:06.614832 6730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:07:06.620702 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 6730
I20260619 01:07:06.621021 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:07:06.621537 388 external_mini_cluster.cc:1428] Setting key 78fe711dd41b7fc74e08678a39e653ad
Jun 19 01:07:06 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:06.638345 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:06.617568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40177 (local address 127.0.97.62:40437)
0619 01:07:06.617996 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:06.618001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:06.619352 (+ 1351us) server_negotiation.cc:407] Connection header received
0619 01:07:06.621595 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:06.621602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:06.621708 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:06.621843 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:06.623721 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:06.624772 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:06.626823 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:06.627558 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:06.627710 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:06.632209 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:06.632215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:06.632218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:06.632254 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:06.634936 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:06.635809 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:06.635814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:06.635817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:06.635947 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:06.636439 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:06.636449 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:06.636451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:06.636718 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:06.636870 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:06.637722 (+ 852us) server_negotiation.cc:299] Negotiation successful
0619 01:07:06.637952 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":222,"threads_started":1}
I20260619 01:07:06.639487 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:06.618224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.2:40177)
0619 01:07:06.619172 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:06.619221 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:06.621215 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:06.622133 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:06.622147 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:06.622802 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:06.623515 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:06.623543 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:06.624960 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:06.624964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:06.626656 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:06.626671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:06.627803 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:06.627808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:06.628044 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:06.628832 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:06.628864 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:06.631930 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:06.635131 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:06.635141 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:06.635162 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:06.635652 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:06.636077 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:06.636082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:06.636086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:06.636314 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:06.636849 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:06.636858 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:06.637251 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 01:07:06.637619 (+ 368us) client_negotiation.cc:240] Negotiation successful
0619 01:07:06.638202 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":408,"threads_started":1}
I20260619 01:07:06.641189 6858 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40437
I20260619 01:07:06.641558 6858 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:06.642314 6858 heartbeater.cc:511] Master 127.0.97.62:40437 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:07:06.643920 6535 ts_manager.cc:194] Registered new tserver with Master: a6e5e05e6f2a4c08a8454513dbd696a7 (127.0.97.2:42509)
I20260619 01:07:06.644788 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 01:07:06.670841 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:07.010875 6864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:07.011174 6864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:07.011269 6864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:07.018846 6864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:07.018976 6864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:07.019094 6864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:07:07.029438 6864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:07.031081 6864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:07.032333 6864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:07.045919 6869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:07.045951 6872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:07.046679 6870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:07.046932 6864 server_base.cc:1034] running on GCE node
I20260619 01:07:07.047569 6864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:07.048565 6864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:07.049757 6864 hybrid_clock.cc:648] HybridClock initialized: now 1781831227049686 us; error 75 us; skew 500 ppm
I20260619 01:07:07.050050 6864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:07 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:07.053527 6864 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 01:07:07.055017 6864 webserver.cc:460] Webserver started at http://127.0.97.3:40237/ using document root <none> and password file <none>
I20260619 01:07:07.055725 6864 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:07.055828 6864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:07.056098 6864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:07.059250 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "08d1a013d240424c8a0855353feed2f6"
format_stamp: "Formatted at 2026-06-19 01:07:07 on dist-test-slave-5p8w"
server_key: "15180cf7debcd7cf564366cf749e0f09"
server_key_iv: "a1c5c3dd92f66677aade7478fdfde7e3"
server_key_version: "encryptionkey@0"
I20260619 01:07:07.060063 6864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "08d1a013d240424c8a0855353feed2f6"
format_stamp: "Formatted at 2026-06-19 01:07:07 on dist-test-slave-5p8w"
server_key: "15180cf7debcd7cf564366cf749e0f09"
server_key_iv: "a1c5c3dd92f66677aade7478fdfde7e3"
server_key_version: "encryptionkey@0"
I20260619 01:07:07.065639 6864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 01:07:07.069262 6879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:07.071012 6864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:07.071213 6864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "08d1a013d240424c8a0855353feed2f6"
format_stamp: "Formatted at 2026-06-19 01:07:07 on dist-test-slave-5p8w"
server_key: "15180cf7debcd7cf564366cf749e0f09"
server_key_iv: "a1c5c3dd92f66677aade7478fdfde7e3"
server_key_version: "encryptionkey@0"
I20260619 01:07:07.071436 6864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:07.110224 6864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:07.114784 6724 heartbeater.cc:503] Master 127.0.97.62:40437 was elected leader, sending a full tablet report...
I20260619 01:07:07.115253 6864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:07.115590 6864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:07.116855 6864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:07.118360 6864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:07.118507 6864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:07.118664 6864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:07.118733 6864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:07.158816 6864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41499
I20260619 01:07:07.158864 6991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41499 every 8 connection(s)
I20260619 01:07:07.160183 6864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:07:07.162443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 6864
I20260619 01:07:07.162645 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:07:07.163197 388 external_mini_cluster.cc:1428] Setting key 3f3226ddf496fde57c694ce55eb42523
Jun 19 01:07:07 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:07.181700 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.164249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57473 (local address 127.0.97.62:40437)
0619 01:07:07.164707 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:07.164716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:07.164736 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:07:07.166933 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:07.166937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:07.167011 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:07.167105 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:07.168671 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.169544 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.171164 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.171573 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.171669 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.175460 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:07.175465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:07.175469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:07.175503 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:07.178637 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.179511 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.179515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.179518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.179636 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.180002 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.180005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.180007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.180261 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:07.180365 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:07.181187 (+ 822us) server_negotiation.cc:299] Negotiation successful
0619 01:07:07.181431 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":652,"threads_started":1}
I20260619 01:07:07.182741 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.163706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.3:57473)
0619 01:07:07.164556 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:07.164609 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:07.166677 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:07.167329 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:07.167337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:07.167879 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:07.168517 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.168536 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.169715 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.169719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.171030 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.171042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.171795 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.171800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.172043 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.172792 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:07.172816 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:07.175155 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:07.178777 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.178784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.178800 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.179402 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.179732 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.179735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.179738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.179914 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.180371 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:07.180377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:07.180770 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 01:07:07.181091 (+ 321us) client_negotiation.cc:240] Negotiation successful
0619 01:07:07.181584 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":334,"threads_started":1}
I20260619 01:07:07.184242 6992 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40437
I20260619 01:07:07.184574 6992 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:07.185377 6992 heartbeater.cc:511] Master 127.0.97.62:40437 requested a full tablet report, sending...
I20260619 01:07:07.186829 6535 ts_manager.cc:194] Registered new tserver with Master: 08d1a013d240424c8a0855353feed2f6 (127.0.97.3:41499)
I20260619 01:07:07.187589 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57473
I20260619 01:07:07.191426 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260619 01:07:07.240850 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44201
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40437,127.0.97.61:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:07.581115 6998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:07.581602 6998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:07.581776 6998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:07.589259 6998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:07:07.589488 6998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:07.589596 6998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:07.589699 6998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:07:07.589792 6998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:07:07.600091 6998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40437,127.0.97.61:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
W20260619 01:07:07.600524 6998 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:40437,127.0.97.61:44201'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 01:07:07.601687 6998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:07.603025 6998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:07.614974 7006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:07.614925 7003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:07.615806 7004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:07.615880 6998 server_base.cc:1034] running on GCE node
I20260619 01:07:07.616707 6998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:07.618341 6998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:07.619688 6998 hybrid_clock.cc:648] HybridClock initialized: now 1781831227619604 us; error 68 us; skew 500 ppm
I20260619 01:07:07.620070 6998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:07 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:07.623437 6998 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 01:07:07.624984 6998 webserver.cc:460] Webserver started at http://127.0.97.61:39361/ using document root <none> and password file <none>
I20260619 01:07:07.625777 6998 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:07.625907 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:07.626245 6998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:07.629473 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data/instance:
uuid: "467da69d46374637abe0bfa08bd696fb"
format_stamp: "Formatted at 2026-06-19 01:07:07 on dist-test-slave-5p8w"
server_key: "1a5ce263a505b78385ef1f3d8fe03e2f"
server_key_iv: "2e5fcd0d47f5990a401bfaa546aa15e1"
server_key_version: "encryptionkey@0"
I20260619 01:07:07.630390 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/wal/instance:
uuid: "467da69d46374637abe0bfa08bd696fb"
format_stamp: "Formatted at 2026-06-19 01:07:07 on dist-test-slave-5p8w"
server_key: "1a5ce263a505b78385ef1f3d8fe03e2f"
server_key_iv: "2e5fcd0d47f5990a401bfaa546aa15e1"
server_key_version: "encryptionkey@0"
I20260619 01:07:07.635672 6998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:07:07.639350 7013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:07.640753 6998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:07:07.640961 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/wal
uuid: "467da69d46374637abe0bfa08bd696fb"
format_stamp: "Formatted at 2026-06-19 01:07:07 on dist-test-slave-5p8w"
server_key: "1a5ce263a505b78385ef1f3d8fe03e2f"
server_key_iv: "2e5fcd0d47f5990a401bfaa546aa15e1"
server_key_version: "encryptionkey@0"
I20260619 01:07:07.641201 6998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:07.647663 6858 heartbeater.cc:503] Master 127.0.97.62:40437 was elected leader, sending a full tablet report...
I20260619 01:07:07.670466 6998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:07.674625 6998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:07.674857 6998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:07.693743 6998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44201
I20260619 01:07:07.693763 7064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44201 every 8 connection(s)
I20260619 01:07:07.695236 6998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
I20260619 01:07:07.699040 7065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:07.702217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 6998
I20260619 01:07:07.707450 7065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:07:07 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 01:07:07 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:07.727334 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.710077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.0.97.62:40437)
0619 01:07:07.710680 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:07.710687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:07.711245 (+ 558us) server_negotiation.cc:407] Connection header received
0619 01:07:07.711422 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:07.711426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:07.711492 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:07.711585 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:07.712915 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.713933 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.715664 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.716077 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.716220 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.720458 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:07.720473 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:07.720478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:07.720528 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:07.723847 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.724810 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.724815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.724819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.724988 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.725489 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.725493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.725496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.725888 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:07.726039 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:07.726740 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 01:07:07.726978 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":291,"threads_started":1}
I20260619 01:07:07.728376 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.704713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.0.97.61:44201)
0619 01:07:07.705624 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:07.705637 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:07.705681 (+ 44us) server_negotiation.cc:407] Connection header received
0619 01:07:07.707350 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:07.707375 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:07.707837 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:07.708312 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:07.709919 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.712068 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.713709 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.714730 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.714921 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.719947 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:07.719956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:07.719971 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:07.720028 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:07.723804 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.724805 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.724815 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.724818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.724989 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.725460 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.725466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.725470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.726174 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:07.726383 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:07.726792 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 01:07:07.727160 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":667,"thread_start_us":380,"threads_started":1}
I20260619 01:07:07.728425 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.710070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48432)
0619 01:07:07.711071 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:07.711123 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:07.711296 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:07.711677 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:07.711685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:07.712510 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:07.712757 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.712765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.714059 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.714064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.715511 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.715521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.716321 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.716330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.716504 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.717153 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:07.717177 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:07.720214 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:07.723993 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.724006 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.724021 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.724687 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.725144 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.725148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.725151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.725381 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.726009 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:07.726015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:07.726312 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 01:07:07.726649 (+ 337us) client_negotiation.cc:240] Negotiation successful
0619 01:07:07.727366 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":236,"thread_start_us":525,"threads_started":1}
I20260619 01:07:07.730329 7065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:07:07 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 01:07:07.742717 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.731156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44201 (local address 127.0.0.1:56394)
0619 01:07:07.731350 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:07.731368 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:07.731463 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:07.731785 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:07.731788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:07.732143 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:07.732279 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.732287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.733330 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.733337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.734206 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.734216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.734884 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.734889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.735098 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.735610 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:07.735627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:07.737681 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:07.740344 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.740349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.740353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.740718 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.741067 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.741074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.741079 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.741205 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.741736 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:07.741740 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:07.741889 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 01:07:07.742045 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 01:07:07.742341 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 01:07:07.742770 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.731429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56394 (local address 127.0.97.61:44201)
0619 01:07:07.731575 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:07.731580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:07.731594 (+ 14us) server_negotiation.cc:407] Connection header received
0619 01:07:07.731633 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:07.731636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:07.731679 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:07.731772 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:07.732415 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.733179 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.734353 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.734752 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.734885 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.737871 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:07.737876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:07.737880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:07.737906 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:07.740193 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.740834 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.740841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.740844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.740962 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.741320 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.741327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.741332 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.741628 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:07.741772 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:07.742116 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 01:07:07.742463 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":161,"server-negotiator.queue_time_us":36}
I20260619 01:07:07.747768 7065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: Bootstrap starting.
I20260619 01:07:07.750932 7065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:07.751950 7065 log.cc:824] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:07.754874 7065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: No bootstrap required, opened a new log
I20260619 01:07:07.761997 7065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } }
I20260619 01:07:07.762356 7065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:07.762435 7065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467da69d46374637abe0bfa08bd696fb, State: Initialized, Role: FOLLOWER
I20260619 01:07:07.763000 7065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } }
I20260619 01:07:07.763832 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } } }
I20260619 01:07:07.764252 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:07:07.764921 7065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:07:07.765344 7072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:07:07.765617 7072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } }
I20260619 01:07:07.767288 7072 leader_election.cc:288] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437)
I20260619 01:07:07.768296 6545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "467da69d46374637abe0bfa08bd696fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" is_pre_election: true
I20260619 01:07:07.769515 7014 leader_election.cc:302] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 467da69d46374637abe0bfa08bd696fb; no voters: 7cf28be23db947b1be7b7bc45f6e54a2
I20260619 01:07:07.770152 7072 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260619 01:07:07.775310 7083 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:07:07.775473 7083 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:07:07.777238 6998 master_runner.cc:428] Detected that this master 467da69d46374637abe0bfa08bd696fb is joining an existing cluster
I20260619 01:07:07.777408 6998 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:44201
I20260619 01:07:07.799677 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.781918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.0.97.62:40437)
0619 01:07:07.782132 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:07.782136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:07.783074 (+ 938us) server_negotiation.cc:407] Connection header received
0619 01:07:07.783341 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:07.783347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:07.783472 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:07.783563 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:07.785279 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.786541 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.789228 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.790045 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.790163 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.792979 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:07.792986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:07.792993 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:07.793035 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:07.796182 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.797078 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.797084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.797087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.797238 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.797777 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.797783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.797789 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.798186 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:07.798301 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:07.798979 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 01:07:07.799255 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 01:07:07.802043 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.781385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48444)
0619 01:07:07.782976 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:07.782997 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:07.783187 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:07.783801 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:07.783806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:07.784607 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:07.785078 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.785094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.787091 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.787098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.789040 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.789059 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.790503 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.790513 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.790779 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.791524 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:07.791552 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:07.792776 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:07.796326 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.796331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.796335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.796925 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.797445 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.797453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.797456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.797638 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.798382 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:07.798389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:07.798648 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 01:07:07.800311 (+ 1663us) client_negotiation.cc:240] Negotiation successful
0619 01:07:07.801509 (+ 1198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1439,"thread_start_us":1265,"threads_started":1}
I20260619 01:07:07.804845 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.782120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.0.97.61:44201)
0619 01:07:07.782406 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:07.782412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:07.789582 (+ 7170us) server_negotiation.cc:407] Connection header received
0619 01:07:07.790041 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:07.790047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:07.790113 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:07.790183 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:07.792054 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.793181 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.794955 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.795704 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.795836 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.798303 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:07.798309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:07.798316 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:07.798353 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:07.801907 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.802746 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.802751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.802753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.802902 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.803293 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.803298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.803301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.803660 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:07.803799 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:07.804274 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 01:07:07.804493 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260619 01:07:07.806008 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.781546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44201 (local address 127.0.0.1:56404)
0619 01:07:07.789466 (+ 7920us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:07.789489 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:07.789906 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:07.790373 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:07.790377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:07.791481 (+ 1104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:07.791882 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.791896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.793418 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.793423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.794808 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.794822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.796143 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.796150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.796329 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.796943 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:07.796970 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:07.798126 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:07.802094 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.802102 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.802108 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.802636 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.803032 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.803035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.803038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.803182 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.803868 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:07.803874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:07.804030 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 01:07:07.805095 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0619 01:07:07.805408 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7769,"thread_start_us":8036,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 19 01:07:07 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831225, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 01:07:07.832301 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.809900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.0.97.61:44201)
0619 01:07:07.810139 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:07.810147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:07.810823 (+ 676us) server_negotiation.cc:407] Connection header received
0619 01:07:07.811185 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:07.811191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:07.817521 (+ 6330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:07.817673 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:07.819150 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.820052 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.821614 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.822099 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.822241 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.826009 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:07.826014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:07.826017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:07.826053 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:07.829013 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.829999 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.830006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.830009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.830157 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.830634 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.830638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.830640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.830921 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:07.831179 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:07.831686 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 01:07:07.831911 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:07:07.832350 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.809865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44201 (local address 127.0.0.1:56420)
0619 01:07:07.810674 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:07.810727 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:07.810917 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:07.817735 (+ 6818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:07.817742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:07.818539 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:07.818861 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.818873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.820207 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.820212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.821434 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.821449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.822277 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.822283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.822504 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.823230 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:07.823249 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:07.825773 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:07.829233 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.829252 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.829266 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.829870 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.830305 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.830312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.830315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.830473 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.831153 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:07.831162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:07.831374 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 01:07:07.831560 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 01:07:07.831912 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":359,"threads_started":1}
I20260619 01:07:07.834586 6535 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:44201: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "467da69d46374637abe0bfa08bd696fb"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 44201
}
attrs {
promote: true
}
}
dest_uuid: "7cf28be23db947b1be7b7bc45f6e54a2"
cas_config_opid_index: -1
I20260619 01:07:07.835570 6535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } }
I20260619 01:07:07.838636 7096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 467da69d46374637abe0bfa08bd696fb (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } }
I20260619 01:07:07.840260 7040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:07.842941 7096 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:44201
I20260619 01:07:07.843851 6998 master.cc:537] Master@127.0.97.61:44201 shutting down...
I20260619 01:07:07.844022 7040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Refusing update from remote peer 7cf28be23db947b1be7b7bc45f6e54a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 01:07:07.843842 7095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } }
I20260619 01:07:07.844204 7095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:07.847533 7096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260619 01:07:07.850659 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37751
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40437,127.0.97.61:44201,127.0.97.60:37751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:07.852512 6521 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 01:07:07.859547 6998 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:07.860308 6998 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:07.860647 6998 tablet_replica.cc:330] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: stopping tablet replica
W20260619 01:07:07.863972 6521 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 -> Peer 467da69d46374637abe0bfa08bd696fb (127.0.97.61:44201): Couldn't send request to peer 467da69d46374637abe0bfa08bd696fb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 01:07:07.868101 6998 master.cc:552] Master@127.0.97.61:44201 shutdown complete.
I20260619 01:07:07.868206 6998 master_runner.cc:305] Clearing existing system tablet
I20260619 01:07:07.868695 6998 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:07.868798 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:07.871958 6998 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 01:07:07.873759 7101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:07.874138 6998 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:07.874440 6998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 01:07:07.874567 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/wal
uuid: "467da69d46374637abe0bfa08bd696fb"
format_stamp: "Formatted at 2026-06-19 01:07:07 on dist-test-slave-5p8w"
server_key: "1a5ce263a505b78385ef1f3d8fe03e2f"
server_key_iv: "2e5fcd0d47f5990a401bfaa546aa15e1"
server_key_version: "encryptionkey@0"
I20260619 01:07:07.879786 6998 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:07.887207 6998 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 01:07:07.887420 6998 log.cc:1196] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 01:07:07.887895 6998 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: Deleting consensus metadata
I20260619 01:07:07.888208 6998 master_runner.cc:315] Copying system tablet from 127.0.97.62:40437
I20260619 01:07:07.890851 6998 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:40437
I20260619 01:07:07.907245 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.891722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.0.97.62:40437)
0619 01:07:07.891905 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:07.891912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:07.892420 (+ 508us) server_negotiation.cc:407] Connection header received
0619 01:07:07.892600 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:07.892606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:07.892705 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:07.892819 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:07.893881 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.895028 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.896842 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.897570 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.897700 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.900079 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:07.900087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:07.900092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:07.900134 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:07.903787 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.904743 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.904750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.904754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.904911 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:07.905456 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:07.905464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:07.905469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:07.905952 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:07.906135 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:07.906614 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 01:07:07.906873 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:07:07.907331 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:07.891624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48450)
0619 01:07:07.892277 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:07.892302 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:07.892463 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:07.892872 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:07.892877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:07.893419 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:07.893715 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.893728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.895193 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.895202 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.896675 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:07.896692 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:07.897740 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:07.897746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:07.897930 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:07.898704 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:07.898727 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:07.899805 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:07.903986 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.903992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.903996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.904591 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.905043 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:07.905047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:07.905050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:07.905244 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:07.906100 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:07.906105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:07.906298 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 01:07:07.906464 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 01:07:07.906832 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":402,"threads_started":1}
I20260619 01:07:07.908591 6555 tablet_copy_service.cc:140] P 7cf28be23db947b1be7b7bc45f6e54a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 467da69d46374637abe0bfa08bd696fb ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:48450)
I20260619 01:07:07.908846 6555 tablet_copy_service.cc:161] P 7cf28be23db947b1be7b7bc45f6e54a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 467da69d46374637abe0bfa08bd696fb at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:48450: session id = 467da69d46374637abe0bfa08bd696fb-00000000000000000000000000000000
I20260619 01:07:07.914109 6555 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 01:07:07.917142 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 01:07:07.924794 6998 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: tablet copy: Starting download of 0 data blocks...
I20260619 01:07:07.925176 6998 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: tablet copy: Starting download of 1 WAL segments...
I20260619 01:07:07.928393 6998 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 01:07:07.933549 6555 tablet_copy_service.cc:342] P 7cf28be23db947b1be7b7bc45f6e54a2: Request end of tablet copy session 467da69d46374637abe0bfa08bd696fb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:48450
I20260619 01:07:07.933770 6555 tablet_copy_service.cc:433] P 7cf28be23db947b1be7b7bc45f6e54a2: ending tablet copy session 467da69d46374637abe0bfa08bd696fb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 467da69d46374637abe0bfa08bd696fb
I20260619 01:07:07.946725 6998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:07.949572 7108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:07.950984 7109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:07.955480 7111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:07.956084 6998 server_base.cc:1034] running on GCE node
I20260619 01:07:07.956446 6998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:07.957131 6998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:07.958391 6998 hybrid_clock.cc:648] HybridClock initialized: now 1781831227958268 us; error 93 us; skew 500 ppm
I20260619 01:07:07.958679 6998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:07 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831227, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:07.960939 6998 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 01:07:07.966344 6998 webserver.cc:460] Webserver started at http://127.0.97.61:41033/ using document root <none> and password file <none>
I20260619 01:07:07.966791 6998 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:07.966914 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:07.970362 6998 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 01:07:07.972704 7118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:07.973220 6998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 01:07:07.973462 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/wal
uuid: "467da69d46374637abe0bfa08bd696fb"
format_stamp: "Formatted at 2026-06-19 01:07:07 on dist-test-slave-5p8w"
server_key: "1a5ce263a505b78385ef1f3d8fe03e2f"
server_key_iv: "2e5fcd0d47f5990a401bfaa546aa15e1"
server_key_version: "encryptionkey@0"
I20260619 01:07:07.973691 6998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:07.996307 6998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:08.000799 6998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:08.035969 6998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44201
I20260619 01:07:08.036006 7169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44201 every 8 connection(s)
I20260619 01:07:08.037194 6998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
I20260619 01:07:08.042418 7170 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:07:08.046478 7170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: Bootstrap starting.
I20260619 01:07:08.063045 7170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:07:08.063530 7170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: Bootstrap complete.
I20260619 01:07:08.064509 7170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } }
I20260619 01:07:08.064751 7170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 467da69d46374637abe0bfa08bd696fb, State: Initialized, Role: LEARNER
I20260619 01:07:08.064970 7170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } }
I20260619 01:07:08.066228 7170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:07:08.065963 7173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } }
I20260619 01:07:08.066542 7173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: This master's current role is: LEARNER
I20260619 01:07:08.078168 7184 catalog_manager.cc:1259] Loaded cluster ID: 6ef0b99fb5d24edda85c4a6401c5bfc3
I20260619 01:07:08.078313 7184 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: loading cluster ID for follower catalog manager: success
I20260619 01:07:08.084461 7184 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: acquiring CA information for follower catalog manager: success
I20260619 01:07:08.087260 7184 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 01:07:08.190418 6992 heartbeater.cc:503] Master 127.0.97.62:40437 was elected leader, sending a full tablet report...
W20260619 01:07:08.271292 7099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:08.271588 7099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:08.271752 7099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:08.279754 7099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:07:08.279901 7099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:08.279981 7099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:08.280076 7099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:07:08.280153 7099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:07:08.290382 7099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40437,127.0.97.61:44201,127.0.97.60:37751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:08.291791 7099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:08.293097 7099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:08.306331 7189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:08.306295 7190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:08.306363 7192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:08.308663 7099 server_base.cc:1034] running on GCE node
I20260619 01:07:08.309196 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.300990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44201 (local address 127.0.0.1:56432)
0619 01:07:08.301167 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:08.301194 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:08.301343 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:08.302013 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:08.302018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:08.302047 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:08.302237 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.302246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.303563 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.303569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.305807 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.305829 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.308379 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.308385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.308575 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.308637 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 01:07:08.308749 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 01:07:08.308904 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 01:07:08.309219 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.301157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.0.97.61:44201)
0619 01:07:08.301685 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:08.301689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:08.301707 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:07:08.301762 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:08.301765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:08.301831 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:08.301962 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:08.302388 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.303419 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.306099 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.307341 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.307427 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.307611 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:08.308781 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0619 01:07:08.308918 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":292,"threads_started":1}
I20260619 01:07:08.309796 7099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:08.310443 7145 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 01:07:08.311041 7173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cf28be23db947b1be7b7bc45f6e54a2. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } }
I20260619 01:07:08.311295 7173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: This master's current role is: LEARNER
I20260619 01:07:08.311503 7099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:08.312727 7099 hybrid_clock.cc:648] HybridClock initialized: now 1781831228312685 us; error 40 us; skew 500 ppm
I20260619 01:07:08.313081 7099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:08 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:08.316227 7099 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 01:07:08.317762 7099 webserver.cc:460] Webserver started at http://127.0.97.60:35331/ using document root <none> and password file <none>
I20260619 01:07:08.318503 7099 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:08.318642 7099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:08.318959 7099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:08.321988 7099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data/instance:
uuid: "5bb91bdb1d574a71a7cf8932a3895367"
format_stamp: "Formatted at 2026-06-19 01:07:08 on dist-test-slave-5p8w"
server_key: "fd496016b2454a1b592927e4eec53faa"
server_key_iv: "cae4428d86139f3ed71ba417200ad689"
server_key_version: "encryptionkey@0"
I20260619 01:07:08.322784 7099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/wal/instance:
uuid: "5bb91bdb1d574a71a7cf8932a3895367"
format_stamp: "Formatted at 2026-06-19 01:07:08 on dist-test-slave-5p8w"
server_key: "fd496016b2454a1b592927e4eec53faa"
server_key_iv: "cae4428d86139f3ed71ba417200ad689"
server_key_version: "encryptionkey@0"
I20260619 01:07:08.328063 7099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:07:08.332331 7200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:08.334050 7099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:07:08.334254 7099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/wal
uuid: "5bb91bdb1d574a71a7cf8932a3895367"
format_stamp: "Formatted at 2026-06-19 01:07:08 on dist-test-slave-5p8w"
server_key: "fd496016b2454a1b592927e4eec53faa"
server_key_iv: "cae4428d86139f3ed71ba417200ad689"
server_key_version: "encryptionkey@0"
I20260619 01:07:08.334456 7099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:08.365607 7099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:08.370151 7099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:08.370441 7099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:08.392180 7099 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37751
I20260619 01:07:08.392198 7251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37751 every 8 connection(s)
I20260619 01:07:08.393811 7099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
I20260619 01:07:08.397891 7252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:08.399662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 7099
I20260619 01:07:08.408313 7252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:07:08 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 01:07:08 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:08.426568 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.401962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.0.97.60:37751)
0619 01:07:08.402835 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:08.402856 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:08.402920 (+ 64us) server_negotiation.cc:407] Connection header received
0619 01:07:08.405004 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:08.405041 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:08.405679 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:08.406195 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:08.407542 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.409709 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.411079 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.411990 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.412299 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.417853 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:08.417861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:08.417899 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:08.417958 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:08.421425 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.422377 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.422385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.422393 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.422569 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.422986 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.422994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.422997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.423722 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:08.424033 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:08.424543 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 01:07:08.425069 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":343,"threads_started":1}
I20260619 01:07:08.428782 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.411673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:47582)
0619 01:07:08.412525 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:08.412577 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:08.412723 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:08.413675 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:08.413683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:08.414633 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:08.415004 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.415031 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.416089 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.416096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.417797 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.417814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.418854 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.418863 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.419093 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.419778 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:08.419805 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:08.422519 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:08.426128 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.426137 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.426143 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.426655 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.427108 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.427113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.427116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.427301 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.427812 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:08.427819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:08.428013 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 01:07:08.428236 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 01:07:08.428489 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":413,"threads_started":1}
I20260619 01:07:08.428956 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.411771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.0.97.62:40437)
0619 01:07:08.413191 (+ 1420us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:08.413196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:08.413210 (+ 14us) server_negotiation.cc:407] Connection header received
0619 01:07:08.413293 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:08.413301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:08.413414 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:08.413526 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:08.415213 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.415902 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.417962 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.418608 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.418779 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.422737 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:08.422742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:08.422746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:08.422780 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:08.425965 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.426832 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.426837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.426841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.426995 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.427409 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.427412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.427414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.427685 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:08.427795 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:08.428372 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 01:07:08.428619 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1315,"thread_start_us":1212,"threads_started":1}
I20260619 01:07:08.430197 7252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:07:08 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 01:07:08.442631 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.430866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44201 (local address 127.0.0.1:44414)
0619 01:07:08.431076 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:08.431096 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:08.431222 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:08.431506 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:08.431509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:08.431807 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:08.431941 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.431948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.432910 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.432914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.434389 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.434403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.435063 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.435068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.435201 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.435771 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:08.435787 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:08.437778 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:08.440644 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.440648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.440651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.441018 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.441386 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.441390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.441392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.441514 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.441951 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:08.441954 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:08.442069 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 01:07:08.442167 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 01:07:08.442404 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 01:07:08.442662 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.430993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.0.97.61:44201)
0619 01:07:08.431179 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:08.431183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:08.431200 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:07:08.431332 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:08.431335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:08.431393 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:08.431475 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:08.432063 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.432764 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.434534 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.434932 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.435043 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.438010 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:08.438021 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:08.438024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:08.438057 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:08.440522 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.441114 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.441117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.441119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.441232 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.441602 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.441605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.441607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.441856 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:08.441964 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:08.442261 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 01:07:08.442435 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 01:07:08.443848 7252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:07:08 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 01:07:08.456377 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.444692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.0.97.60:37751)
0619 01:07:08.444842 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:08.444848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:08.444865 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:07:08.445011 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:08.445016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:08.445080 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:08.445176 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:08.445868 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.446765 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.447842 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.448241 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.448377 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.451328 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:08.451339 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:08.451342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:08.451368 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:08.453923 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.454526 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.454531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.454536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.454646 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.455012 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.455016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.455019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.455291 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:08.455442 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:08.455760 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 01:07:08.456073 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":55}
I20260619 01:07:08.456375 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.444555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37751 (local address 127.0.0.1:48532)
0619 01:07:08.444775 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:08.444795 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:08.444905 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:08.445189 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:08.445193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:08.445596 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:08.445737 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.445745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.446882 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.446890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.447711 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.447721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.448392 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.448397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.448593 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.449113 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:08.449128 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:08.451157 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:08.454058 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.454064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.454068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.454403 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.454748 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.454755 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.454758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.454900 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.455403 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:08.455408 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:08.455541 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 01:07:08.455674 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 01:07:08.455949 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 01:07:08.461817 7252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: Bootstrap starting.
I20260619 01:07:08.464982 7252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:08.466174 7252 log.cc:824] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:08.469039 7252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: No bootstrap required, opened a new log
I20260619 01:07:08.476526 7252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } }
I20260619 01:07:08.476866 7252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:08.477005 7252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb91bdb1d574a71a7cf8932a3895367, State: Initialized, Role: FOLLOWER
I20260619 01:07:08.477713 7252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } }
I20260619 01:07:08.479008 7259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } } }
I20260619 01:07:08.479377 7259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:07:08.479740 7252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 01:07:08.490025 7270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:07:08.490175 7270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:07:08.493948 7099 master_runner.cc:428] Detected that this master 5bb91bdb1d574a71a7cf8932a3895367 is joining an existing cluster
I20260619 01:07:08.494102 7099 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:37751
I20260619 01:07:08.524087 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.502903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.0.97.62:40437)
0619 01:07:08.503110 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:08.503118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:08.504517 (+ 1399us) server_negotiation.cc:407] Connection header received
0619 01:07:08.504795 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:08.504803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:08.504892 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:08.504976 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:08.506913 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.508449 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.511775 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.512697 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.512828 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.516469 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:08.516480 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:08.516487 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:08.516543 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:08.520387 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.521040 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.521047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.521050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.521208 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.521711 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.521716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.521720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.522059 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:08.522189 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:08.523517 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0619 01:07:08.523737 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 01:07:08.524190 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.502903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:47588)
0619 01:07:08.503931 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:08.503952 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:08.504637 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:08.505212 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:08.505220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:08.506180 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:08.506674 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.506692 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.508731 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.508743 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.511523 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.511546 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.513225 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.513234 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.513527 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.514538 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:08.514573 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:08.516150 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:08.520516 (+ 4366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.520523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.520530 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.520900 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.521368 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.521372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.521376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.521570 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.522245 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:08.522249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:08.523141 (+ 892us) client_negotiation.cc:769] Sending connection context
0619 01:07:08.523315 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 01:07:08.523731 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":72,"thread_start_us":15324,"threads_started":1}
I20260619 01:07:08.536530 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.505136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.0.97.60:37751)
0619 01:07:08.505397 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:08.505406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:08.518083 (+ 12677us) server_negotiation.cc:407] Connection header received
0619 01:07:08.518277 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:08.518283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:08.518370 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:08.518477 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:08.521149 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.522708 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.525420 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.525846 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.526012 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.528959 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:08.528966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:08.528971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:08.529007 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:08.533342 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.534188 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.534194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.534197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.534333 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.534746 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.534750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.534753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.535173 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:08.535342 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:08.535885 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 01:07:08.536132 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 01:07:08.537658 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.503159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44418 (local address 127.0.97.61:44201)
0619 01:07:08.503328 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:08.503334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:08.518091 (+ 14757us) server_negotiation.cc:407] Connection header received
0619 01:07:08.518370 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:08.518376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:08.518466 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:08.518553 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:08.520280 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.521404 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.523937 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.524588 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.524694 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.529269 (+ 4575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:08.529279 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:08.529311 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:08.529355 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:08.533651 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.534698 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.534707 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.534713 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.534898 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.535494 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.535502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.535507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.535941 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:08.536094 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:08.536890 (+ 796us) server_negotiation.cc:299] Negotiation successful
0619 01:07:08.537179 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 01:07:08.539321 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.503242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44201 (local address 127.0.0.1:44418)
0619 01:07:08.517948 (+ 14706us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:08.517972 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:08.518211 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:08.518766 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:08.518771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:08.519654 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:08.520085 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.520098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.521608 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.521615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.523748 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.523762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.526462 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.526471 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.526641 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.527240 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:08.527262 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:08.529022 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:08.533851 (+ 4829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.533859 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.533865 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.534515 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.535069 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.535076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.535082 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.535287 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.536216 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:08.536223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:08.536516 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 01:07:08.538042 (+ 1526us) client_negotiation.cc:240] Negotiation successful
0619 01:07:08.538518 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14511,"mutex_wait_us":27,"spinlock_wait_cycles":28544,"thread_start_us":573,"threads_started":1}
I20260619 01:07:08.542526 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.504697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37751 (local address 127.0.0.1:48544)
0619 01:07:08.517997 (+ 13300us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:08.518023 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:08.518149 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:08.518656 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:08.518664 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:08.520014 (+ 1350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:08.520459 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.521038 (+ 579us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.522945 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.522950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.525251 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.525263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.526066 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.526078 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.526254 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.527317 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:08.527342 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:08.528715 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:08.533530 (+ 4815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.533535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.533539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.534059 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.534471 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.534476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.534479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.534628 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.535429 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:08.535434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:08.535628 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 01:07:08.541487 (+ 5859us) client_negotiation.cc:240] Negotiation successful
0619 01:07:08.541947 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13196,"mutex_wait_us":3,"spinlock_wait_cycles":51584,"thread_start_us":12788,"threads_started":1}
I20260619 01:07:08.544390 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 6596
Jun 19 01:07:08 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831225, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 01:07:08.555368 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.530626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37751 (local address 127.0.0.1:48546)
0619 01:07:08.531014 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:08.531035 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:08.531149 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:08.537321 (+ 6172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:08.537327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:08.537861 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:08.538092 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.538112 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.539947 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.539953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.541400 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.541419 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.542573 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.542582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.544117 (+ 1535us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.544823 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:08.544847 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:08.548543 (+ 3696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:08.552554 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.552559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.552564 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.553120 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.553593 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.553598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.553602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.553734 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.554273 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:08.554278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:08.554478 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 01:07:08.554632 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 01:07:08.554925 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260619 01:07:08.555503 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.531376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.0.97.60:37751)
0619 01:07:08.536868 (+ 5492us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:08.536876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:08.536893 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:07:08.537000 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:08.537006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:08.537102 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:08.538281 (+ 1179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:08.538467 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.539768 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.541595 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.542369 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.544484 (+ 2115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.548759 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:08.548766 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:08.548771 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:08.548810 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:08.552397 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.553258 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.553263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.553267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.553461 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.553851 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.553856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.553859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.554145 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:08.554292 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:08.554738 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 01:07:08.554956 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5392,"thread_start_us":11685,"threads_started":1}
Jun 19 01:07:08 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 01:07:08.557586 6535 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:37751: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 37751
}
attrs {
promote: true
}
}
dest_uuid: "7cf28be23db947b1be7b7bc45f6e54a2"
cas_config_opid_index: 5
I20260619 01:07:08.558687 6535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } }
I20260619 01:07:08.560381 7095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5bb91bdb1d574a71a7cf8932a3895367 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:08.562325 7145 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 LEARNER]: Refusing update from remote peer 7cf28be23db947b1be7b7bc45f6e54a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 01:07:08.563313 7227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:08.568571 7227 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 FOLLOWER]: Refusing update from remote peer 7cf28be23db947b1be7b7bc45f6e54a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 01:07:08.569420 7281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 01:07:08.570832 7283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260619 01:07:08.574116 7145 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5bb91bdb1d574a71a7cf8932a3895367 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:08.575227 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.545424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.0.97.1:45717)
0619 01:07:08.546023 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:08.546033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:08.546063 (+ 30us) server_negotiation.cc:407] Connection header received
0619 01:07:08.546133 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:08.546140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:08.546358 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:08.546542 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:08.547861 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.549054 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.551025 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.552051 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.552193 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.563639 (+ 11446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:08.563662 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:08.563681 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:08.563729 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:08.569458 (+ 5729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.570840 (+ 1382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.570854 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.570860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.571066 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.571756 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.571765 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.571770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.572368 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:08.572780 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:08.574365 (+ 1585us) server_negotiation.cc:299] Negotiation successful
0619 01:07:08.574664 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":425,"threads_started":1}
I20260619 01:07:08.578294 7173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:08.578563 7173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: This master's current role is: LEARNER
I20260619 01:07:08.578917 6719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:08.579137 7226 raft_consensus.cc:858] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } }
W20260619 01:07:08.579581 7226 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } }
I20260619 01:07:08.581426 7095 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:37751
I20260619 01:07:08.582180 7099 master.cc:537] Master@127.0.97.60:37751 shutting down...
I20260619 01:07:08.582216 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:08.582552 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:08.588521 7282 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2: attempting to promote NON_VOTER 467da69d46374637abe0bfa08bd696fb to VOTER
I20260619 01:07:08.589425 7282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } }
I20260619 01:07:08.592907 7145 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 LEARNER]: Refusing update from remote peer 7cf28be23db947b1be7b7bc45f6e54a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 01:07:08.598044 7282 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 01:07:08.598942 7282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:08.599491 7282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: This master's current role is: LEADER
W20260619 01:07:08.601351 6521 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 -> Peer 5bb91bdb1d574a71a7cf8932a3895367 (127.0.97.60:37751): Couldn't send request to peer 5bb91bdb1d574a71a7cf8932a3895367. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 01:07:08.601823 7282 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 467da69d46374637abe0bfa08bd696fb (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:08.606863 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:08.607158 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:08.606757 7145 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 467da69d46374637abe0bfa08bd696fb (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:08.606537 7282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:08.607659 7282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:08.611578 7173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:08.611894 7173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:07:08.627178 7226 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 467da69d46374637abe0bfa08bd696fb (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } }
I20260619 01:07:08.636096 7291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } }
I20260619 01:07:08.636471 7291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 01:07:08.636826 7259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cf28be23db947b1be7b7bc45f6e54a2. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } }
I20260619 01:07:08.637036 7259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 01:07:08.637470 7294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: true } } }
I20260619 01:07:08.637706 7294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 01:07:08.639227 7099 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 01:07:08.639722 7099 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 01:07:08.639905 7099 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: stopping tablet replica
I20260619 01:07:08.658875 7099 master.cc:552] Master@127.0.97.60:37751 shutdown complete.
I20260619 01:07:08.658987 7099 master_runner.cc:305] Clearing existing system tablet
I20260619 01:07:08.659379 7099 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:08.659487 7099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:08.662503 7099 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 01:07:08.664222 7296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:08.664625 7099 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:08.664867 7099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 01:07:08.664965 7099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/wal
uuid: "5bb91bdb1d574a71a7cf8932a3895367"
format_stamp: "Formatted at 2026-06-19 01:07:08 on dist-test-slave-5p8w"
server_key: "fd496016b2454a1b592927e4eec53faa"
server_key_iv: "cae4428d86139f3ed71ba417200ad689"
server_key_version: "encryptionkey@0"
I20260619 01:07:08.670113 7099 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:08.677927 7099 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 01:07:08.678129 7099 log.cc:1196] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 01:07:08.678664 7099 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: Deleting consensus metadata
I20260619 01:07:08.678920 7099 master_runner.cc:315] Copying system tablet from 127.0.97.62:40437
I20260619 01:07:08.683010 7099 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:40437
I20260619 01:07:08.699754 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.683895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.0.97.62:40437)
0619 01:07:08.684132 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:08.684139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:08.684646 (+ 507us) server_negotiation.cc:407] Connection header received
0619 01:07:08.684820 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:08.684826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:08.684921 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:08.685033 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:08.686112 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.687260 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.689223 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.689937 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.690079 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.692466 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:08.692481 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:08.692486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:08.692527 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:08.696591 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.697529 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.697535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.697539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.697690 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:08.698142 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:08.698147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:08.698151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:08.698521 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:08.698666 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:08.699133 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 01:07:08.699358 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 01:07:08.700151 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:08.683859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:47590)
0619 01:07:08.684511 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:08.684536 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:08.684691 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:08.685086 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:08.685091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:08.685667 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:08.685919 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.685934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.687450 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.687456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.689051 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:08.689067 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:08.690125 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:08.690131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:08.690312 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:08.691110 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:08.691136 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:08.692235 (+ 1099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:08.696754 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.696760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.696764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.697373 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.697821 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:08.697826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:08.697830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:08.698024 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:08.698653 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:08.698658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:08.698851 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 01:07:08.699019 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 01:07:08.699357 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":392,"threads_started":1}
I20260619 01:07:08.700789 6555 tablet_copy_service.cc:140] P 7cf28be23db947b1be7b7bc45f6e54a2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5bb91bdb1d574a71a7cf8932a3895367 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:47590)
I20260619 01:07:08.700961 6555 tablet_copy_service.cc:161] P 7cf28be23db947b1be7b7bc45f6e54a2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5bb91bdb1d574a71a7cf8932a3895367 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:47590: session id = 5bb91bdb1d574a71a7cf8932a3895367-00000000000000000000000000000000
I20260619 01:07:08.704538 6555 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 01:07:08.706465 7099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 01:07:08.713927 7099 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: tablet copy: Starting download of 0 data blocks...
I20260619 01:07:08.714258 7099 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: tablet copy: Starting download of 1 WAL segments...
I20260619 01:07:08.716881 7099 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 01:07:08.722625 6555 tablet_copy_service.cc:342] P 7cf28be23db947b1be7b7bc45f6e54a2: Request end of tablet copy session 5bb91bdb1d574a71a7cf8932a3895367-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:47590
I20260619 01:07:08.722788 6555 tablet_copy_service.cc:433] P 7cf28be23db947b1be7b7bc45f6e54a2: ending tablet copy session 5bb91bdb1d574a71a7cf8932a3895367-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5bb91bdb1d574a71a7cf8932a3895367
I20260619 01:07:08.723387 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 6596
I20260619 01:07:08.735901 7099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:08.738950 7303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:08.739835 7304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:08.747437 7306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:08.748100 7099 server_base.cc:1034] running on GCE node
I20260619 01:07:08.748594 7099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:08.749343 7099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:08.750216 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45717
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40437
--webserver_port=35289
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:40437,127.0.97.61:44201
--tserver_master_addrs=127.0.97.62:40437,127.0.97.61:44201,127.0.97.60:37751 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:07:08.750631 7099 hybrid_clock.cc:648] HybridClock initialized: now 1781831228750571 us; error 51 us; skew 500 ppm
I20260619 01:07:08.750931 7099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:08 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831228, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:08.753369 7099 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 01:07:08.758765 7099 webserver.cc:460] Webserver started at http://127.0.97.60:35383/ using document root <none> and password file <none>
I20260619 01:07:08.759105 7099 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:08.759220 7099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:08.762315 7099 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 01:07:08.764549 7314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:08.765029 7099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 01:07:08.765203 7099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/wal
uuid: "5bb91bdb1d574a71a7cf8932a3895367"
format_stamp: "Formatted at 2026-06-19 01:07:08 on dist-test-slave-5p8w"
server_key: "fd496016b2454a1b592927e4eec53faa"
server_key_iv: "cae4428d86139f3ed71ba417200ad689"
server_key_version: "encryptionkey@0"
I20260619 01:07:08.765475 7099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:08.782907 7099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:08.787387 7099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:08.806038 7099 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37751
I20260619 01:07:08.806097 7365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37751 every 8 connection(s)
I20260619 01:07:08.807292 7099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
I20260619 01:07:08.812950 7366 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:07:08.816915 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: Bootstrap starting.
I20260619 01:07:08.833873 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:07:08.834458 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: Bootstrap complete.
I20260619 01:07:08.835662 7366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } }
I20260619 01:07:08.835969 7366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5bb91bdb1d574a71a7cf8932a3895367, State: Initialized, Role: LEARNER
I20260619 01:07:08.836210 7366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } }
I20260619 01:07:08.837111 7369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:08.837460 7366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:07:08.837607 7369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: This master's current role is: LEARNER
I20260619 01:07:08.849483 7380 catalog_manager.cc:1259] Loaded cluster ID: 6ef0b99fb5d24edda85c4a6401c5bfc3
I20260619 01:07:08.849628 7380 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: loading cluster ID for follower catalog manager: success
I20260619 01:07:08.853266 7380 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: acquiring CA information for follower catalog manager: success
I20260619 01:07:08.855950 7380 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 01:07:09.112197 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.104332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37751 (local address 127.0.0.1:48556)
0619 01:07:09.104871 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:09.104896 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:09.105054 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:09.105634 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:09.105637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:09.105660 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:09.105840 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.105847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.107744 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.107749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.109503 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.109521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.111417 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.111423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.111565 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.111597 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:07:09.111706 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 01:07:09.111848 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":247,"threads_started":1}
I20260619 01:07:09.112262 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.104418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.0.97.60:37751)
0619 01:07:09.105211 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:09.105219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:09.105246 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:07:09.105360 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:09.105365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:09.105466 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:09.105619 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:09.105989 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.107524 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.109710 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.111186 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.111354 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.111632 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:09.111755 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 01:07:09.111902 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":488,"threads_started":1}
I20260619 01:07:09.113327 7341 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 01:07:09.113763 7369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cf28be23db947b1be7b7bc45f6e54a2. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:09.114068 7369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: This master's current role is: LEARNER
W20260619 01:07:09.144243 7312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:09.144553 7312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:09.144747 7312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:09.152886 7312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:09.153086 7312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:09.153249 7312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:07:09.163439 7312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35289
--tserver_master_addrs=127.0.97.62:40437,127.0.97.61:44201,127.0.97.60:37751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:09.164916 7312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:09.166540 7312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:09.181319 7387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:09.181348 7390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:09.181275 7388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:09.182082 7312 server_base.cc:1034] running on GCE node
I20260619 01:07:09.182803 7312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:09.183825 7312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:09.185091 7312 hybrid_clock.cc:648] HybridClock initialized: now 1781831229185020 us; error 60 us; skew 500 ppm
I20260619 01:07:09.185518 7312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:09 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:09.189461 7312 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 01:07:09.191108 7312 webserver.cc:460] Webserver started at http://127.0.97.1:35289/ using document root <none> and password file <none>
I20260619 01:07:09.191825 7312 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:09.191944 7312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:09.198292 7312 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:07:09.201468 7397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:09.203019 7312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:07:09.203230 7312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "bd4233d57aad4aa991a36d8abcd4913a"
format_stamp: "Formatted at 2026-06-19 01:07:05 on dist-test-slave-5p8w"
server_key: "5e33be9871e1972ea274b77382e61440"
server_key_iv: "ee167aaa84a81009adac73be3eb06855"
server_key_version: "encryptionkey@0"
I20260619 01:07:09.204315 7312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:09.219538 7312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:09.224284 7312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:09.224642 7312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:09.225768 7312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:09.227380 7312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:09.227473 7312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:09.227643 7312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:09.227744 7312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:09.264524 7312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45717
I20260619 01:07:09.264581 7509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45717 every 8 connection(s)
I20260619 01:07:09.266237 7312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:07:09.270212 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 7312
I20260619 01:07:09.270534 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 6730
Jun 19 01:07:09 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 01:07:09 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Jun 19 01:07:09 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 01:07:09 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 01:07:09.299448 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.271076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33319 (local address 127.0.97.62:40437)
0619 01:07:09.276101 (+ 5025us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:09.276107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:09.276127 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:07:09.276206 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:09.276214 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:09.276342 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:09.279612 (+ 3270us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:09.279825 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.281312 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.283988 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.284415 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.284564 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.291556 (+ 6992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:09.291565 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:09.291571 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:09.291615 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:09.295288 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.296503 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.296511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.296517 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.296664 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.297263 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.297271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.297277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.297641 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:09.297746 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:09.298700 (+ 954us) server_negotiation.cc:299] Negotiation successful
0619 01:07:09.298986 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4859,"thread_start_us":4767,"threads_started":1}
I20260619 01:07:09.300513 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.270452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.1:33319)
0619 01:07:09.271453 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:09.271514 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:09.274625 (+ 3111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:09.276973 (+ 2348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:09.276993 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:09.278130 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:09.279315 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.279351 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.282272 (+ 2921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.282278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.283800 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.283813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.284687 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.284697 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.285140 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.286402 (+ 1262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:09.286445 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:09.291261 (+ 4816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:09.295486 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.295496 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.295518 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.296351 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.296821 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.296826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.296829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.297104 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.297822 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:09.297828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:09.298234 (+ 406us) client_negotiation.cc:769] Sending connection context
0619 01:07:09.298531 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 01:07:09.299008 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":653,"threads_started":1}
I20260619 01:07:09.302146 7511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40437
I20260619 01:07:09.302490 7511 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:09.303462 7511 heartbeater.cc:511] Master 127.0.97.62:40437 requested a full tablet report, sending...
I20260619 01:07:09.305982 6535 ts_manager.cc:194] Re-registered known tserver with Master: bd4233d57aad4aa991a36d8abcd4913a (127.0.97.1:45717)
I20260619 01:07:09.307166 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33319
I20260619 01:07:09.308678 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.282314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53857 (local address 127.0.97.60:37751)
0619 01:07:09.282503 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:09.282509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:09.288546 (+ 6037us) server_negotiation.cc:407] Connection header received
0619 01:07:09.288759 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:09.288765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:09.288842 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:09.288937 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:09.290403 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.291426 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.293422 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.294093 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.294231 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.300556 (+ 6325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:09.300562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:09.300574 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:09.300615 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:09.305067 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.306473 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.306479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.306482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.306644 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.307103 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.307109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.307113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.307428 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:09.307558 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:09.308104 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 01:07:09.308342 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 01:07:09.308871 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.272201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.0.97.2:42509)
0619 01:07:09.275695 (+ 3494us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:09.275708 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:09.281822 (+ 6114us) server_negotiation.cc:407] Connection header received
0619 01:07:09.283227 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:09.283242 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:09.283607 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:09.283796 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:09.286988 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.289093 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.291302 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.292278 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.292385 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.300955 (+ 8570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:09.300977 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:09.300997 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:09.301042 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:09.304967 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.306058 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.306071 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.306076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.306259 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.306768 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.306773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.306776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.307221 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:09.307523 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:09.308131 (+ 608us) server_negotiation.cc:299] Negotiation successful
0619 01:07:09.308443 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3249,"thread_start_us":3106,"threads_started":1}
I20260619 01:07:09.309513 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.270453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44201 (local address 127.0.97.1:32967)
0619 01:07:09.281920 (+ 11467us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:09.281946 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:09.282084 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:09.287566 (+ 5482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:09.287572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:09.288450 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:09.288827 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.288840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.290765 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.290770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.293274 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.293315 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.294620 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.294629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.294814 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.295454 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:09.295477 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:09.299597 (+ 4120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:09.305510 (+ 5913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.305516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.305520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.306337 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.306769 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.306774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.306778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.306943 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.307708 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:09.307712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:09.307862 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 01:07:09.308020 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 01:07:09.308374 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11293,"mutex_wait_us":862,"spinlock_wait_cycles":148608,"thread_start_us":10932,"threads_started":1}
I20260619 01:07:09.310519 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.271031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32967 (local address 127.0.97.61:44201)
0619 01:07:09.287126 (+ 16095us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:09.287132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:09.287160 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:07:09.287217 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:09.287222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:09.287310 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:09.287652 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:09.288997 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.290602 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.293485 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.294209 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.294346 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.299905 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:09.299912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:09.299917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:09.299969 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:09.305327 (+ 5358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.306485 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.306490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.306495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.306632 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.307073 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.307078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.307081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.307516 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:09.307640 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:09.309968 (+ 2328us) server_negotiation.cc:299] Negotiation successful
0619 01:07:09.310192 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15956,"thread_start_us":16340,"threads_started":1}
I20260619 01:07:09.309825 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.282227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37751 (local address 127.0.97.1:53857)
0619 01:07:09.288401 (+ 6174us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:09.288426 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:09.288623 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:09.289101 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:09.289106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:09.289776 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:09.290079 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.290092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.291596 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.291601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.292921 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.292931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.294486 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.294494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.294736 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.295614 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:09.295641 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:09.300297 (+ 4656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:09.305682 (+ 5385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.305687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.305691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.306323 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.306789 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.306794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.306797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.306978 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.307637 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:09.307641 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:09.307790 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 01:07:09.308941 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0619 01:07:09.309410 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6038,"mutex_wait_us":93,"thread_start_us":5761,"threads_started":1}
I20260619 01:07:09.311149 6853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:09.311708 7510 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37751
I20260619 01:07:09.311939 7510 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:09.311967 7512 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44201
I20260619 01:07:09.312111 7512 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:09.312599 7512 heartbeater.cc:511] Master 127.0.97.61:44201 requested a full tablet report, sending...
I20260619 01:07:09.312636 7510 heartbeater.cc:511] Master 127.0.97.60:37751 requested a full tablet report, sending...
I20260619 01:07:09.314652 7135 ts_manager.cc:194] Registered new tserver with Master: bd4233d57aad4aa991a36d8abcd4913a (127.0.97.1:45717)
I20260619 01:07:09.316143 7331 ts_manager.cc:194] Registered new tserver with Master: bd4233d57aad4aa991a36d8abcd4913a (127.0.97.1:45717)
I20260619 01:07:09.399703 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 6730
I20260619 01:07:09.416164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42509
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40437
--webserver_port=41853
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:40437,127.0.97.61:44201
--tserver_master_addrs=127.0.97.62:40437,127.0.97.61:44201,127.0.97.60:37751 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:07:09.530645 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: true } } }
I20260619 01:07:09.531067 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:09.532768 7523 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2: attempting to promote NON_VOTER 5bb91bdb1d574a71a7cf8932a3895367 to VOTER
I20260619 01:07:09.533413 7523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } }
I20260619 01:07:09.535279 7341 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 LEARNER]: Refusing update from remote peer 7cf28be23db947b1be7b7bc45f6e54a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 01:07:09.535424 7145 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Refusing update from remote peer 7cf28be23db947b1be7b7bc45f6e54a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 01:07:09.535998 7523 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 01:07:09.536393 7524 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 01:07:09.539240 7523 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5bb91bdb1d574a71a7cf8932a3895367 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:09.540153 7145 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5bb91bdb1d574a71a7cf8932a3895367 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:09.543581 7523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:09.543923 7523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:09.544219 7523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:09.545015 7523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:09.547036 7528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:09.547430 7528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:07:09.547778 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:09.548056 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:09.548539 7341 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5bb91bdb1d574a71a7cf8932a3895367 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:09.551317 7369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:09.551633 7369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 01:07:09.767755 7522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:09.768132 7522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:09.768338 7522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:09.776324 7522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:09.776487 7522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:09.776677 7522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:07:09.787024 7522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41853
--tserver_master_addrs=127.0.97.62:40437,127.0.97.61:44201,127.0.97.60:37751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:09.788631 7522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:09.790143 7522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:09.804919 7533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:09.805001 7536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:09.804977 7534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:09.811198 7522 server_base.cc:1034] running on GCE node
I20260619 01:07:09.812100 7522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:09.813333 7522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:09.814616 7522 hybrid_clock.cc:648] HybridClock initialized: now 1781831229814571 us; error 38 us; skew 500 ppm
I20260619 01:07:09.814999 7522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:09 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:09.818533 7522 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 01:07:09.820147 7522 webserver.cc:460] Webserver started at http://127.0.97.2:41853/ using document root <none> and password file <none>
I20260619 01:07:09.820875 7522 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:09.821004 7522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:09.827154 7522 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 01:07:09.830243 7543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:09.831825 7522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:09.831969 7522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "a6e5e05e6f2a4c08a8454513dbd696a7"
format_stamp: "Formatted at 2026-06-19 01:07:06 on dist-test-slave-5p8w"
server_key: "52d45b37fe3155ed64224da013cc7987"
server_key_iv: "2858254bebec73bc82795a9df140a719"
server_key_version: "encryptionkey@0"
I20260619 01:07:09.832994 7522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:09.854928 7522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:09.861048 7522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:09.861431 7522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:09.862794 7522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:09.864692 7522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:09.864830 7522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:09.864939 7522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:09.865023 7522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:09.905962 7522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42509
I20260619 01:07:09.906020 7655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42509 every 8 connection(s)
I20260619 01:07:09.907529 7522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:07:09.908874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 7522
I20260619 01:07:09.909224 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 6864
Jun 19 01:07:09 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260619 01:07:09.947582 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.910381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.0.97.3:41499)
0619 01:07:09.914307 (+ 3926us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:09.914320 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:09.914362 (+ 42us) server_negotiation.cc:407] Connection header received
0619 01:07:09.914454 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:09.914466 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:09.914818 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:09.919046 (+ 4228us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:09.919389 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.921158 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.926730 (+ 5572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.927968 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.936419 (+ 8451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.936768 (+ 349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:09.936782 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:09.936802 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:09.936843 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:09.941348 (+ 4505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.942428 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.942440 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.942444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.942630 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.943135 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.943140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.943143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.943608 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:09.943916 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:09.946860 (+ 2944us) server_negotiation.cc:299] Negotiation successful
0619 01:07:09.947150 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3715,"thread_start_us":3575,"threads_started":1}
I20260619 01:07:09.948987 6987 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 19 01:07:09 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 01:07:09 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 01:07:09 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831229, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 01:07:09.960464 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.920263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36495 (local address 127.0.97.61:44201)
0619 01:07:09.937394 (+ 17131us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:09.937400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:09.937428 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:07:09.943090 (+ 5662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:09.943097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:09.943209 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:09.943319 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:09.944765 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.945900 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.947927 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.948577 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.948678 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.952773 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:09.952780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:09.952785 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:09.952837 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:09.956483 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.957539 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.957544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.957547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.957686 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.958263 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.958268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.958271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.958634 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:09.958761 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:09.959788 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0619 01:07:09.960026 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16984,"thread_start_us":16939,"threads_started":1}
I20260619 01:07:09.961727 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.920230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44201 (local address 127.0.97.2:36495)
0619 01:07:09.936986 (+ 16756us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:09.937011 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:09.939916 (+ 2905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:09.943504 (+ 3588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:09.943509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:09.944185 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:09.944540 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.944553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.946089 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.946094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.947759 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.947777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.948976 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.948982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.949176 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.950280 (+ 1104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:09.950311 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:09.952579 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:09.956671 (+ 4092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.956687 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.956703 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.957389 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.957840 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.957844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.957849 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.958131 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.958850 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:09.958858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:09.959317 (+ 459us) client_negotiation.cc:769] Sending connection context
0619 01:07:09.960735 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0619 01:07:09.961015 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16594,"spinlock_wait_cycles":21760,"thread_start_us":12579,"threads_started":1}
I20260619 01:07:09.962375 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.912439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55461 (local address 127.0.97.62:40437)
0619 01:07:09.933427 (+ 20988us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:09.933435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:09.933460 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:07:09.933532 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:09.933538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:09.933637 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:09.933752 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:09.938071 (+ 4319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.939655 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.943130 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.944056 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.944214 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.952498 (+ 8284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:09.952506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:09.952511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:09.952556 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:09.956494 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.958135 (+ 1641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.958141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.958144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.958308 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.958717 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.958722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.958726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.959059 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:09.959194 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:09.961824 (+ 2630us) server_negotiation.cc:299] Negotiation successful
0619 01:07:09.962050 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20812,"thread_start_us":31929,"threads_started":1}
I20260619 01:07:09.962683 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.912410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.2:55461)
0619 01:07:09.932466 (+ 20056us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:09.932493 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:09.932659 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:09.934286 (+ 1627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:09.934305 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:09.935369 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:09.936377 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.936405 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.939906 (+ 3501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.939913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.942528 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.942548 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.945321 (+ 2773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.945331 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.946011 (+ 680us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.948132 (+ 2121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:09.948168 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:09.952179 (+ 4011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:09.957402 (+ 5223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.957406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.957410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.957980 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.958443 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.958448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.958451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.958583 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.959264 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:09.959268 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:09.959393 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 01:07:09.959665 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 01:07:09.960227 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19863,"mutex_wait_us":1,"thread_start_us":24275,"threads_started":1}
I20260619 01:07:09.963436 7658 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44201
I20260619 01:07:09.963462 7657 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40437
I20260619 01:07:09.963523 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.911533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37751 (local address 127.0.97.2:38915)
0619 01:07:09.912599 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:09.912653 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:09.916287 (+ 3634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:09.940164 (+ 23877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:09.940173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:09.940829 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:09.941261 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.941280 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.946132 (+ 4852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.946144 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.947218 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:09.947230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.949223 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.949232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:09.949993 (+ 761us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.950805 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:09.950832 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:09.954225 (+ 3393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:09.959344 (+ 5119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.959352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.959358 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.960098 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.960625 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:09.960632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:09.960638 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:09.960799 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:09.961629 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:09.961637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:09.961849 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 01:07:09.962524 (+ 675us) client_negotiation.cc:240] Negotiation successful
0619 01:07:09.962865 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"spinlock_wait_cycles":34432,"thread_start_us":4202,"threads_started":1}
I20260619 01:07:09.963814 7658 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:09.963812 7657 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:09.965234 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:09.912883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38915 (local address 127.0.97.60:37751)
0619 01:07:09.937601 (+ 24718us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:09.937609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:09.937631 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:07:09.937710 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:09.937718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:09.937821 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:09.941511 (+ 3690us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:09.941777 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.945071 (+ 3294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.947383 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:09.948956 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:09.950431 (+ 1475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:09.954525 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:09.954534 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:09.954542 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:09.954595 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:09.959139 (+ 4544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.960306 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.960314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.960321 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.960495 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:09.960988 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:09.960996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:09.961002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:09.961465 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:09.963837 (+ 2372us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:09.963997 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 01:07:09.964254 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24577,"thread_start_us":24548,"threads_started":1}
I20260619 01:07:09.965356 7657 heartbeater.cc:511] Master 127.0.97.62:40437 requested a full tablet report, sending...
I20260619 01:07:09.966338 7656 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37751
I20260619 01:07:09.966454 7656 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:09.967229 6535 ts_manager.cc:194] Re-registered known tserver with Master: a6e5e05e6f2a4c08a8454513dbd696a7 (127.0.97.2:42509)
I20260619 01:07:09.968568 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55461
I20260619 01:07:09.969388 7658 heartbeater.cc:511] Master 127.0.97.61:44201 requested a full tablet report, sending...
I20260619 01:07:09.969424 7656 heartbeater.cc:511] Master 127.0.97.60:37751 requested a full tablet report, sending...
I20260619 01:07:09.970665 7135 ts_manager.cc:194] Registered new tserver with Master: a6e5e05e6f2a4c08a8454513dbd696a7 (127.0.97.2:42509)
I20260619 01:07:09.971021 7331 ts_manager.cc:194] Registered new tserver with Master: a6e5e05e6f2a4c08a8454513dbd696a7 (127.0.97.2:42509)
I20260619 01:07:10.055264 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 6864
I20260619 01:07:10.071923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41499
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40437
--webserver_port=40237
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:40437,127.0.97.61:44201
--tserver_master_addrs=127.0.97.62:40437,127.0.97.61:44201,127.0.97.60:37751 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:07:10.310997 7511 heartbeater.cc:503] Master 127.0.97.62:40437 was elected leader, sending a full tablet report...
W20260619 01:07:10.424950 7669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:10.425289 7669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:10.425532 7669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:10.433638 7669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:10.433768 7669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:10.433955 7669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:07:10.444418 7669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40237
--tserver_master_addrs=127.0.97.62:40437,127.0.97.61:44201,127.0.97.60:37751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:10.445932 7669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:10.447376 7669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:10.461133 7674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:10.461184 7675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:10.461578 7677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:10.462393 7669 server_base.cc:1034] running on GCE node
I20260619 01:07:10.463136 7669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:10.464170 7669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:10.465411 7669 hybrid_clock.cc:648] HybridClock initialized: now 1781831230465333 us; error 54 us; skew 500 ppm
I20260619 01:07:10.465801 7669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:10 dist-test-slave-5p8w krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831230, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:10.469635 7669 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 01:07:10.471762 7669 webserver.cc:460] Webserver started at http://127.0.97.3:40237/ using document root <none> and password file <none>
I20260619 01:07:10.472604 7669 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:10.472765 7669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:10.479038 7669 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 01:07:10.482455 7684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:10.484270 7669 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 01:07:10.484490 7669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "08d1a013d240424c8a0855353feed2f6"
format_stamp: "Formatted at 2026-06-19 01:07:07 on dist-test-slave-5p8w"
server_key: "15180cf7debcd7cf564366cf749e0f09"
server_key_iv: "a1c5c3dd92f66677aade7478fdfde7e3"
server_key_version: "encryptionkey@0"
I20260619 01:07:10.485698 7669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:10.521426 7669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:10.526695 7669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:10.526997 7669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:10.528095 7669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:10.529680 7669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:10.529803 7669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:10.529937 7669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:10.530040 7669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:10.568439 7669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41499
I20260619 01:07:10.568491 7796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41499 every 8 connection(s)
I20260619 01:07:10.570165 7669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:07:10.575646 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 7669
Jun 19 01:07:10 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831230, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 01:07:10 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831230, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 01:07:10.600642 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.573141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54881 (local address 127.0.97.62:40437)
0619 01:07:10.573619 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:10.573624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:10.577385 (+ 3761us) server_negotiation.cc:407] Connection header received
0619 01:07:10.577461 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:10.577467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:10.577555 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:10.577697 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:10.579349 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.580553 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.583426 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.584263 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.584395 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.591715 (+ 7320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:10.591723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:10.591727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:10.591776 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:10.595779 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.596962 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.596970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.596976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.597167 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.597868 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.597875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.597880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.598315 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:10.598500 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:10.599767 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0619 01:07:10.600086 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":263,"threads_started":1}
I20260619 01:07:10.603320 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.573973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37751 (local address 127.0.97.3:52041)
0619 01:07:10.581649 (+ 7676us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:10.581669 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:10.581796 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:10.582331 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:10.582337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:10.582988 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:10.584558 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.584570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.586098 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.586103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.587846 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.587869 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.589343 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.589354 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.589725 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.590510 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:10.590531 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:10.593131 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:10.598010 (+ 4879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:10.598018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:10.598024 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:10.598686 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:10.599238 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:10.599245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:10.599250 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:10.599497 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:10.600266 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:10.600270 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:10.600458 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 01:07:10.600649 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 01:07:10.601421 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7527,"mutex_wait_us":3,"spinlock_wait_cycles":31104,"thread_start_us":7201,"threads_started":1}
I20260619 01:07:10.604773 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.574105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52041 (local address 127.0.97.60:37751)
0619 01:07:10.581075 (+ 6970us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:10.581081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:10.581902 (+ 821us) server_negotiation.cc:407] Connection header received
0619 01:07:10.581978 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:10.581983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:10.582066 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:10.582167 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:10.584737 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.585907 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.588117 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.588935 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.589066 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.593425 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:10.593435 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:10.593442 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:10.593498 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:10.597806 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.598876 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.598885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.598891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.599085 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.599652 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.599660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.599666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.600057 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:10.600203 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:10.604015 (+ 3812us) server_negotiation.cc:299] Negotiation successful
0619 01:07:10.604343 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6824,"thread_start_us":10984,"threads_started":1}
I20260619 01:07:10.605542 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.573899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.3:54881)
0619 01:07:10.574888 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:10.574947 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:10.576760 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:10.577958 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:10.577968 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:10.578523 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:10.579129 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.579146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.580720 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.580728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.583270 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.583283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.584528 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.584536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.584880 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.585761 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:10.585791 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:10.590522 (+ 4731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:10.596026 (+ 5504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:10.596041 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:10.596072 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:10.596801 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:10.597383 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:10.597391 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:10.597397 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:10.597728 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:10.598622 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:10.598634 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:10.599225 (+ 591us) client_negotiation.cc:769] Sending connection context
0619 01:07:10.601310 (+ 2085us) client_negotiation.cc:240] Negotiation successful
0619 01:07:10.601720 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":422,"threads_started":1}
I20260619 01:07:10.606562 7797 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37751
I20260619 01:07:10.606969 7797 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:10.608011 7797 heartbeater.cc:511] Master 127.0.97.60:37751 requested a full tablet report, sending...
I20260619 01:07:10.609170 7798 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40437
I20260619 01:07:10.609340 7798 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:10.609812 7798 heartbeater.cc:511] Master 127.0.97.62:40437 requested a full tablet report, sending...
I20260619 01:07:10.610782 6535 ts_manager.cc:194] Re-registered known tserver with Master: 08d1a013d240424c8a0855353feed2f6 (127.0.97.3:41499)
I20260619 01:07:10.610795 7331 ts_manager.cc:194] Registered new tserver with Master: 08d1a013d240424c8a0855353feed2f6 (127.0.97.3:41499)
I20260619 01:07:10.611925 6535 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54881
Jun 19 01:07:10 dist-test-slave-5p8w krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831230, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 01:07:10.635584 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.581983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44201 (local address 127.0.97.3:60779)
0619 01:07:10.589219 (+ 7236us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:10.589244 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:10.589623 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:10.610158 (+ 20535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:10.610167 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:10.611036 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:10.611441 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.611458 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.616682 (+ 5224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.616690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.618468 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.618487 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.622029 (+ 3542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.622037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.622217 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.623231 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:10.623256 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:10.626839 (+ 3583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:10.632283 (+ 5444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:10.632292 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:10.632297 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:10.632839 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:10.633639 (+ 800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:10.633647 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:10.633652 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:10.633827 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:10.634445 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:10.634450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:10.634629 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 01:07:10.634774 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 01:07:10.635126 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7031,"thread_start_us":7336,"threads_started":1}
I20260619 01:07:10.635705 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.583635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.0.97.62:40437)
0619 01:07:10.593398 (+ 9763us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:10.593405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:10.613347 (+ 19942us) server_negotiation.cc:407] Connection header received
0619 01:07:10.613411 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:10.613416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:10.613508 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:10.614122 (+ 614us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:10.618606 (+ 4484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.619773 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.622018 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.622675 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.622772 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.627368 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:10.627377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:10.627385 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:10.627437 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:10.631117 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.632021 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.632027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.632032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.632184 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.633870 (+ 1686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.633875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.633879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.634339 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:10.634567 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:10.635057 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 01:07:10.635303 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6708,"thread_start_us":7048,"threads_started":1}
I20260619 01:07:10.636560 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.613248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.0.97.60:37751)
0619 01:07:10.614897 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:10.614906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:10.615188 (+ 282us) server_negotiation.cc:407] Connection header received
0619 01:07:10.615443 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:10.615451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:10.615594 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:10.615674 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:10.618679 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.619377 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.621477 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.622160 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.622269 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.629606 (+ 7337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:10.629613 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:10.629618 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:10.629658 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:10.632987 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.634236 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.634242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.634246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.634428 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.634904 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.634909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.634924 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.635242 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:10.635455 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:10.636024 (+ 569us) server_negotiation.cc:299] Negotiation successful
0619 01:07:10.636249 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1511}
I20260619 01:07:10.636948 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.582541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60779 (local address 127.0.97.61:44201)
0619 01:07:10.609780 (+ 27239us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:10.609789 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:10.609814 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:07:10.609879 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:10.609884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:10.609962 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:10.610090 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:10.614620 (+ 4530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.616093 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.619964 (+ 3871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.620632 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.620759 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.627941 (+ 7182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:10.627950 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:10.627974 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:10.628020 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:10.632104 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.632999 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.633005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.633009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.633152 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.633959 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.633966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.633972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.634293 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:10.634441 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:10.635556 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0619 01:07:10.636252 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27094,"thread_start_us":26982,"threads_started":1}
I20260619 01:07:10.638815 7799 heartbeater.cc:347] Connected to a master server at 127.0.97.61:44201
I20260619 01:07:10.638988 7799 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:10.639462 7799 heartbeater.cc:511] Master 127.0.97.61:44201 requested a full tablet report, sending...
I20260619 01:07:10.639603 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.594250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.0.97.61:44201)
0619 01:07:10.613381 (+ 19131us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:10.613387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:10.615026 (+ 1639us) server_negotiation.cc:407] Connection header received
0619 01:07:10.615084 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:10.615089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:10.615157 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:10.615267 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:10.618603 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.619807 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.621889 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.622554 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.622679 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.626978 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:10.626986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:10.626990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:10.627036 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:10.633029 (+ 5993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.636435 (+ 3406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.636442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.636447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.636620 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:10.637128 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:10.637136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:10.637140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:10.637584 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:10.637828 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:10.638966 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0619 01:07:10.639216 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18990,"thread_start_us":20006,"threads_started":1}
I20260619 01:07:10.641901 7134 ts_manager.cc:194] Registered new tserver with Master: 08d1a013d240424c8a0855353feed2f6 (127.0.97.3:41499)
I20260619 01:07:10.640969 6535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:07:10.648638 6535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:07:10.688751 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.670671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.0.97.2:42509)
0619 01:07:10.679129 (+ 8458us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:10.679137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:10.680802 (+ 1665us) server_negotiation.cc:407] Connection header received
0619 01:07:10.681147 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:10.681156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:10.681430 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:10.681618 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:10.682698 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.684284 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.686743 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.687661 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.687827 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.688082 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:10.688269 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 01:07:10.688438 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8280,"thread_start_us":8166,"threads_started":1}
I20260619 01:07:10.688833 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.669776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42509 (local address 127.0.0.1:49918)
0619 01:07:10.680676 (+ 10900us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:10.680693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:10.680910 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:10.682149 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:10.682156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:10.682181 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:10.682519 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.682532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.684438 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.684441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.686585 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.686596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.687849 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.687856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.688060 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.688096 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 01:07:10.688220 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 01:07:10.688380 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10799,"mutex_wait_us":1,"thread_start_us":9740,"threads_started":1}
I20260619 01:07:10.689370 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.669357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.0.97.3:41499)
0619 01:07:10.678035 (+ 8678us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:10.678049 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:10.679730 (+ 1681us) server_negotiation.cc:407] Connection header received
0619 01:07:10.679852 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:10.679864 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:10.680189 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:10.680425 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:10.681935 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.683025 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.686280 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.687805 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.687920 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.688151 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:10.688613 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 01:07:10.689007 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8415,"thread_start_us":8200,"threads_started":1}
I20260619 01:07:10.689340 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.668963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41499 (local address 127.0.0.1:42288)
0619 01:07:10.679344 (+ 10381us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:10.679369 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:10.679573 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:10.681148 (+ 1575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:10.681152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:10.681174 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:10.681719 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.681733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.683173 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.683181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.685609 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.685628 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.688299 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.688304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.688420 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.688442 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:07:10.688554 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 01:07:10.688695 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10237,"mutex_wait_us":78,"thread_start_us":11719,"threads_started":1}
I20260619 01:07:10.689966 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.669381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45717 (local address 127.0.0.1:44710)
0619 01:07:10.680084 (+ 10703us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:10.680101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:10.680230 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:10.680782 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:10.680786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:10.680813 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:10.681253 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.681266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.683635 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.683643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.686020 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:10.686034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.688990 (+ 2956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.688998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:10.689195 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.689225 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 01:07:10.689380 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 01:07:10.689553 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10571,"mutex_wait_us":44,"thread_start_us":9765,"threads_started":1}
I20260619 01:07:10.693187 7591 tablet_service.cc:1467] Processing CreateTablet for tablet d78c6b5b6a6047a88cf193f3b746c5cc (DEFAULT_TABLE table=test-table [id=8f77e2d6724349f8856cbc3dfeeff154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:10.693185 7732 tablet_service.cc:1467] Processing CreateTablet for tablet d78c6b5b6a6047a88cf193f3b746c5cc (DEFAULT_TABLE table=test-table [id=8f77e2d6724349f8856cbc3dfeeff154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:10.694588 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78c6b5b6a6047a88cf193f3b746c5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:10.694936 7732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78c6b5b6a6047a88cf193f3b746c5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:10.701637 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:10.670262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.0.97.1:45717)
0619 01:07:10.678665 (+ 8403us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:10.678678 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:10.680225 (+ 1547us) server_negotiation.cc:407] Connection header received
0619 01:07:10.680380 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:10.680386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:10.680584 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:10.680792 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:10.681529 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.683478 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.686207 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:10.687775 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:10.687920 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:10.688151 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:10.690206 (+ 2055us) server_negotiation.cc:299] Negotiation successful
0619 01:07:10.690403 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8167,"thread_start_us":8024,"threads_started":1}
I20260619 01:07:10.705922 7445 tablet_service.cc:1467] Processing CreateTablet for tablet d78c6b5b6a6047a88cf193f3b746c5cc (DEFAULT_TABLE table=test-table [id=8f77e2d6724349f8856cbc3dfeeff154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:10.707538 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d78c6b5b6a6047a88cf193f3b746c5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:10.707835 7832 tablet_bootstrap.cc:492] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: Bootstrap starting.
I20260619 01:07:10.712285 7832 tablet_bootstrap.cc:654] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:10.713462 7832 log.cc:824] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:10.714336 7833 tablet_bootstrap.cc:492] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: Bootstrap starting.
I20260619 01:07:10.716811 7832 tablet_bootstrap.cc:492] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: No bootstrap required, opened a new log
I20260619 01:07:10.717113 7832 ts_tablet_manager.cc:1397] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 01:07:10.718699 7833 tablet_bootstrap.cc:654] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:10.719978 7833 log.cc:824] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:10.723512 7833 tablet_bootstrap.cc:492] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: No bootstrap required, opened a new log
I20260619 01:07:10.723858 7833 ts_tablet_manager.cc:1397] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260619 01:07:10.726228 7836 tablet_bootstrap.cc:492] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: Bootstrap starting.
I20260619 01:07:10.727499 7832 raft_consensus.cc:348] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:10.728035 7832 raft_consensus.cc:374] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:10.728147 7832 raft_consensus.cc:729] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e5e05e6f2a4c08a8454513dbd696a7, State: Initialized, Role: FOLLOWER
I20260619 01:07:10.728854 7832 consensus_queue.cc:260] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:10.730770 7832 ts_tablet_manager.cc:1428] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260619 01:07:10.731221 7836 tablet_bootstrap.cc:654] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:10.731454 7657 heartbeater.cc:503] Master 127.0.97.62:40437 was elected leader, sending a full tablet report...
I20260619 01:07:10.732688 7836 log.cc:824] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:10.738457 7833 raft_consensus.cc:348] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:10.738978 7833 raft_consensus.cc:374] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:10.739084 7833 raft_consensus.cc:729] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d1a013d240424c8a0855353feed2f6, State: Initialized, Role: FOLLOWER
I20260619 01:07:10.739748 7833 consensus_queue.cc:260] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:10.740693 7836 tablet_bootstrap.cc:492] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: No bootstrap required, opened a new log
I20260619 01:07:10.740803 7798 heartbeater.cc:503] Master 127.0.97.62:40437 was elected leader, sending a full tablet report...
I20260619 01:07:10.741008 7836 ts_tablet_manager.cc:1397] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260619 01:07:10.741781 7833 ts_tablet_manager.cc:1428] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: Time spent starting tablet: real 0.018s user 0.010s sys 0.008s
I20260619 01:07:10.749688 7836 raft_consensus.cc:348] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:10.750072 7836 raft_consensus.cc:374] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:10.750144 7836 raft_consensus.cc:729] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4233d57aad4aa991a36d8abcd4913a, State: Initialized, Role: FOLLOWER
I20260619 01:07:10.750677 7836 consensus_queue.cc:260] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:10.752830 7836 ts_tablet_manager.cc:1428] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: Time spent starting tablet: real 0.012s user 0.007s sys 0.006s
W20260619 01:07:10.772861 7513 tablet.cc:2369] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:07:10.823212 7800 tablet.cc:2369] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:07:10.947010 7667 tablet.cc:2369] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:07:11.016189 7837 raft_consensus.cc:482] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:07:11.016610 7837 raft_consensus.cc:504] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:11.018738 7837 leader_election.cc:288] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d1a013d240424c8a0855353feed2f6 (127.0.97.3:41499), bd4233d57aad4aa991a36d8abcd4913a (127.0.97.1:45717)
I20260619 01:07:11.028687 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:11.019151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43355 (local address 127.0.97.3:41499)
0619 01:07:11.019334 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:11.019341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:11.020128 (+ 787us) server_negotiation.cc:407] Connection header received
0619 01:07:11.020316 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:11.020322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:11.020401 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:11.020515 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:11.021430 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.022557 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.025966 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.027278 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.027431 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:11.027539 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:11.028354 (+ 815us) server_negotiation.cc:299] Negotiation successful
0619 01:07:11.028506 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:07:11.028777 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:11.019151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47335 (local address 127.0.97.1:45717)
0619 01:07:11.019360 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:11.019367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:11.020128 (+ 761us) server_negotiation.cc:407] Connection header received
0619 01:07:11.020316 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:11.020322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:11.020401 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:11.020510 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:11.021430 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.022557 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.025974 (+ 3417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.027279 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.027431 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:11.027538 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:11.028384 (+ 846us) server_negotiation.cc:299] Negotiation successful
0619 01:07:11.028531 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 01:07:11.028873 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:11.019048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41499 (local address 127.0.97.2:43355)
0619 01:07:11.020000 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:11.020016 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:11.020172 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:11.020546 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:11.020551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:11.020585 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:11.021220 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:11.021280 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.022739 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.022746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:11.025707 (+ 2961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:11.025732 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.027654 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.027659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:11.027996 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:11.028139 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 01:07:11.028258 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 01:07:11.028435 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":441,"spinlock_wait_cycles":2816,"thread_start_us":464,"threads_started":1}
I20260619 01:07:11.029620 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:11.019148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45717 (local address 127.0.97.2:47335)
0619 01:07:11.019973 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:11.019999 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:11.020172 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:11.020546 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:11.020551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:11.020581 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:11.021220 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:11.021234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.022755 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.022759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:11.025707 (+ 2948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:11.025739 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.027645 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.027653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:11.027997 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:11.028139 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 01:07:11.029054 (+ 915us) client_negotiation.cc:240] Negotiation successful
0619 01:07:11.029220 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":340,"spinlock_wait_cycles":16000,"thread_start_us":421,"threads_started":1}
I20260619 01:07:11.030074 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78c6b5b6a6047a88cf193f3b746c5cc" candidate_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d1a013d240424c8a0855353feed2f6" is_pre_election: true
I20260619 01:07:11.030752 7752 raft_consensus.cc:2454] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e5e05e6f2a4c08a8454513dbd696a7 in term 0.
I20260619 01:07:11.030932 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78c6b5b6a6047a88cf193f3b746c5cc" candidate_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd4233d57aad4aa991a36d8abcd4913a" is_pre_election: true
I20260619 01:07:11.031442 7465 raft_consensus.cc:2454] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e5e05e6f2a4c08a8454513dbd696a7 in term 0.
I20260619 01:07:11.031474 7544 leader_election.cc:302] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d1a013d240424c8a0855353feed2f6, a6e5e05e6f2a4c08a8454513dbd696a7; no voters:
I20260619 01:07:11.031890 7837 raft_consensus.cc:2781] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:07:11.032056 7837 raft_consensus.cc:482] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:07:11.032176 7837 raft_consensus.cc:3037] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:11.034899 7837 raft_consensus.cc:504] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:11.035555 7837 leader_election.cc:288] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [CANDIDATE]: Term 1 election: Requested vote from peers 08d1a013d240424c8a0855353feed2f6 (127.0.97.3:41499), bd4233d57aad4aa991a36d8abcd4913a (127.0.97.1:45717)
I20260619 01:07:11.036152 7465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78c6b5b6a6047a88cf193f3b746c5cc" candidate_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd4233d57aad4aa991a36d8abcd4913a"
I20260619 01:07:11.036177 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d78c6b5b6a6047a88cf193f3b746c5cc" candidate_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d1a013d240424c8a0855353feed2f6"
I20260619 01:07:11.036384 7752 raft_consensus.cc:3037] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:11.036384 7465 raft_consensus.cc:3037] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:11.039773 7752 raft_consensus.cc:2454] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e5e05e6f2a4c08a8454513dbd696a7 in term 1.
I20260619 01:07:11.039773 7465 raft_consensus.cc:2454] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e5e05e6f2a4c08a8454513dbd696a7 in term 1.
I20260619 01:07:11.040369 7547 leader_election.cc:302] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08d1a013d240424c8a0855353feed2f6, a6e5e05e6f2a4c08a8454513dbd696a7, bd4233d57aad4aa991a36d8abcd4913a; no voters:
I20260619 01:07:11.040822 7837 raft_consensus.cc:2781] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:11.041431 7837 raft_consensus.cc:686] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 1 LEADER]: Becoming Leader. State: Replica: a6e5e05e6f2a4c08a8454513dbd696a7, State: Running, Role: LEADER
I20260619 01:07:11.043069 7837 consensus_queue.cc:237] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:11.052811 6535 catalog_manager.cc:5495] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e5e05e6f2a4c08a8454513dbd696a7 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } health_report { overall_health: HEALTHY } } }
I20260619 01:07:11.149042 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:11.143453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.0.97.2:42509)
0619 01:07:11.143630 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:11.143634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:11.144063 (+ 429us) server_negotiation.cc:407] Connection header received
0619 01:07:11.144235 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:11.144239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:11.144318 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:11.144470 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:07:11.144814 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.145614 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.146944 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.147343 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.147450 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:11.147962 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:07:11.148529 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:07:11.148641 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:11.148756 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 01:07:11.148829 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 01:07:11.157848 7465 raft_consensus.cc:1261] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a [term 1 FOLLOWER]: Refusing update from remote peer a6e5e05e6f2a4c08a8454513dbd696a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:11.157850 7752 raft_consensus.cc:1261] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6 [term 1 FOLLOWER]: Refusing update from remote peer a6e5e05e6f2a4c08a8454513dbd696a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:11.158987 7843 consensus_queue.cc:1035] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:11.159425 7837 consensus_queue.cc:1035] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:11.174204 7848 mvcc.cc:205] Tried to move back new op lower bound from 7298380722811224064 to 7298380722356203520. Current Snapshot: MvccSnapshot[applied={T|T < 7298380722811224064}]
I20260619 01:07:11.195324 6535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:11.195657 6535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:11.202100 6535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cf28be23db947b1be7b7bc45f6e54a2: Sending DeleteTablet for 3 replicas of tablet d78c6b5b6a6047a88cf193f3b746c5cc
I20260619 01:07:11.203573 7732 tablet_service.cc:1514] Processing DeleteTablet for tablet d78c6b5b6a6047a88cf193f3b746c5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42288
I20260619 01:07:11.203769 7445 tablet_service.cc:1514] Processing DeleteTablet for tablet d78c6b5b6a6047a88cf193f3b746c5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44710
I20260619 01:07:11.204308 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet d78c6b5b6a6047a88cf193f3b746c5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:11 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49918
I20260619 01:07:11.204649 7857 tablet_replica.cc:330] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: stopping tablet replica
I20260619 01:07:11.205281 7857 raft_consensus.cc:2229] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:11.206051 7857 raft_consensus.cc:2258] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:11.213887 7858 tablet_replica.cc:330] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: stopping tablet replica
I20260619 01:07:11.214560 7858 raft_consensus.cc:2229] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:11.215060 7858 raft_consensus.cc:2258] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:11.215867 7859 tablet_replica.cc:330] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: stopping tablet replica
I20260619 01:07:11.216272 7859 raft_consensus.cc:2229] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:07:11.216718 7857 ts_tablet_manager.cc:1905] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:11.217003 7859 raft_consensus.cc:2258] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:11.219158 7859 ts_tablet_manager.cc:1905] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:11.219158 7858 ts_tablet_manager.cc:1905] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:11.225106 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 6503
I20260619 01:07:11.226400 6565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:11.226570 7858 ts_tablet_manager.cc:1918] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:11.226747 7858 log.cc:1196] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/d78c6b5b6a6047a88cf193f3b746c5cc
I20260619 01:07:11.227239 7859 ts_tablet_manager.cc:1918] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:11.227322 7858 ts_tablet_manager.cc:1939] T d78c6b5b6a6047a88cf193f3b746c5cc P bd4233d57aad4aa991a36d8abcd4913a: Deleting consensus metadata
I20260619 01:07:11.227536 7859 log.cc:1196] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/d78c6b5b6a6047a88cf193f3b746c5cc
I20260619 01:07:11.228154 7859 ts_tablet_manager.cc:1939] T d78c6b5b6a6047a88cf193f3b746c5cc P a6e5e05e6f2a4c08a8454513dbd696a7: Deleting consensus metadata
I20260619 01:07:11.230611 7857 ts_tablet_manager.cc:1918] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:11.230818 7857 log.cc:1196] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/d78c6b5b6a6047a88cf193f3b746c5cc
I20260619 01:07:11.231395 7857 ts_tablet_manager.cc:1939] T d78c6b5b6a6047a88cf193f3b746c5cc P 08d1a013d240424c8a0855353feed2f6: Deleting consensus metadata
I20260619 01:07:11.247118 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:11.207536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.0.97.61:44201)
0619 01:07:11.213965 (+ 6429us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:11.213973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:11.220783 (+ 6810us) server_negotiation.cc:407] Connection header received
0619 01:07:11.221009 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:11.221015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:11.221112 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:11.221229 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:11.232117 (+ 10888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.233231 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.235539 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.236016 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.236129 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:11.239783 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:11.239791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:11.239796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:11.239846 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:11.243598 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:11.244668 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:11.244674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:11.244679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:11.244841 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:11.245350 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:11.245355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:11.245358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:11.245761 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:11.246011 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:11.246551 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 01:07:11.246779 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6265,"thread_start_us":6473,"threads_started":1}
I20260619 01:07:11.268038 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:11.221891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.0.97.60:37751)
0619 01:07:11.233548 (+ 11657us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:11.233556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:11.248322 (+ 14766us) server_negotiation.cc:407] Connection header received
0619 01:07:11.248608 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:11.248614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:11.248705 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:11.248786 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:11.250836 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.252167 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.254897 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.255650 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.255783 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:11.260450 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:11.260459 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:11.260464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:11.260520 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:11.264340 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:11.265332 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:11.265340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:11.265344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:11.265516 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:11.266063 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:11.266069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:11.266073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:11.266453 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:11.266708 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:11.267353 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 01:07:11.267601 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11510,"thread_start_us":11306,"threads_started":1}
I20260619 01:07:11.369236 6520 catalog_manager.cc:4841] TS a6e5e05e6f2a4c08a8454513dbd696a7 (127.0.97.2:42509): tablet d78c6b5b6a6047a88cf193f3b746c5cc (table test-table [id=8f77e2d6724349f8856cbc3dfeeff154]) successfully deleted
I20260619 01:07:11.369141 6519 catalog_manager.cc:4841] TS 08d1a013d240424c8a0855353feed2f6 (127.0.97.3:41499): tablet d78c6b5b6a6047a88cf193f3b746c5cc (table test-table [id=8f77e2d6724349f8856cbc3dfeeff154]) successfully deleted
W20260619 01:07:11.369552 6520 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.2:42509, user_credentials={real_user=slave}} blocked reactor thread for 139609us
W20260619 01:07:11.369557 6519 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.3:41499, user_credentials={real_user=slave}} blocked reactor thread for 136461us
I20260619 01:07:11.369181 6522 catalog_manager.cc:4841] TS bd4233d57aad4aa991a36d8abcd4913a (127.0.97.1:45717): tablet d78c6b5b6a6047a88cf193f3b746c5cc (table test-table [id=8f77e2d6724349f8856cbc3dfeeff154]) successfully deleted
W20260619 01:07:11.369984 6522 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.1:45717, user_credentials={real_user=slave}} blocked reactor thread for 140522us
I20260619 01:07:11.370524 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 6503
I20260619 01:07:11.419080 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:11.397568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.0.97.61:44201)
0619 01:07:11.397853 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:11.397863 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:11.399710 (+ 1847us) server_negotiation.cc:407] Connection header received
0619 01:07:11.400022 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:11.400029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:11.400120 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:11.400199 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:11.403086 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.404230 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.406534 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.407261 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.407375 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:11.411777 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:11.411784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:11.411789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:11.411839 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:11.415351 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:11.416332 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:11.416340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:11.416344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:11.416514 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:11.417088 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:11.417096 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:11.417101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:11.417541 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:11.417825 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:11.418458 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 01:07:11.418715 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 01:07:11.425568 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:11.400436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.0.97.60:37751)
0619 01:07:11.400689 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:11.400698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:11.406686 (+ 5988us) server_negotiation.cc:407] Connection header received
0619 01:07:11.407035 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:11.407045 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:11.407184 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:11.407290 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:11.411433 (+ 4143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.412580 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.414662 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:11.415335 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:11.415483 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:11.419415 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:11.419423 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:11.419428 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:11.419474 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:11.422666 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:11.423435 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:11.423440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:11.423443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:11.423573 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:11.424000 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:11.424009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:11.424011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:11.424306 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:11.424564 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:11.424946 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 01:07:11.425185 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 01:07:12.182061 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.181045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.2:55393)
0619 01:07:12.181665 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.181831 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":369,"threads_started":1}
W20260619 01:07:12.182531 7657 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40437 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
I20260619 01:07:12.185964 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.185042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.1:42373)
0619 01:07:12.185577 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.185692 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":286,"threads_started":1}
W20260619 01:07:12.186437 7511 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40437 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
I20260619 01:07:12.189284 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.188455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.3:45857)
0619 01:07:12.188902 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.189069 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":244,"threads_started":1}
W20260619 01:07:12.189780 7798 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40437 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
I20260619 01:07:12.703300 7874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7cf28be23db947b1be7b7bc45f6e54a2)
I20260619 01:07:12.703616 7874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } }
I20260619 01:07:12.703732 7875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7cf28be23db947b1be7b7bc45f6e54a2)
I20260619 01:07:12.704020 7875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } }
I20260619 01:07:12.704577 7874 leader_election.cc:288] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437), 5bb91bdb1d574a71a7cf8932a3895367 (127.0.97.60:37751)
I20260619 01:07:12.712116 7875 leader_election.cc:288] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437), 467da69d46374637abe0bfa08bd696fb (127.0.97.61:44201)
I20260619 01:07:12.716337 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.705954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48018)
0619 01:07:12.715753 (+ 9799us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.716013 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9659,"threads_started":1}
W20260619 01:07:12.719306 7315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
I20260619 01:07:12.722635 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.704715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48014)
0619 01:07:12.722072 (+ 17357us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.722293 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17147,"threads_started":1}
W20260619 01:07:12.728045 7119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
I20260619 01:07:12.734684 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.714566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44201 (local address 127.0.0.1:44460)
0619 01:07:12.717827 (+ 3261us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.717856 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:12.718031 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:12.724352 (+ 6321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:12.724361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:12.724400 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:12.724788 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:12.724806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:12.727758 (+ 2952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:12.727764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:12.730649 (+ 2885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:12.730684 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:12.733360 (+ 2676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:12.733372 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:12.733649 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:12.733715 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 01:07:12.733824 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 01:07:12.734095 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3116,"thread_start_us":880,"threads_started":1}
I20260619 01:07:12.735396 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.734776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48034)
0619 01:07:12.734976 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.735153 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 01:07:12.735682 7315 leader_election.cc:334] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437): Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
I20260619 01:07:12.735886 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.712991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.0.97.61:44201)
0619 01:07:12.723810 (+ 10819us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:12.723820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:12.723851 (+ 31us) server_negotiation.cc:407] Connection header received
0619 01:07:12.723963 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:12.723973 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:12.724088 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:12.725014 (+ 926us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:12.725393 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:12.727123 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:12.731024 (+ 3901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:12.732987 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:12.735005 (+ 2018us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:12.735115 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:12.735166 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 01:07:12.735343 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10618,"spinlock_wait_cycles":13568,"thread_start_us":10356,"threads_started":1}
I20260619 01:07:12.735983 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.705164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.0.97.60:37751)
0619 01:07:12.723471 (+ 18307us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:12.723481 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:12.724376 (+ 895us) server_negotiation.cc:407] Connection header received
0619 01:07:12.724468 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:12.724477 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:12.724574 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:12.724704 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:12.725786 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:12.727558 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:12.731259 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:12.733229 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:12.733451 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:12.733641 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:12.735028 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0619 01:07:12.735202 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18137,"thread_start_us":17923,"threads_started":1}
I20260619 01:07:12.736110 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.704809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37751 (local address 127.0.0.1:48586)
0619 01:07:12.724008 (+ 19199us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.724037 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:12.724234 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:12.725039 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:12.725047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:12.725086 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:12.725506 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:12.725523 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:12.727810 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:12.727819 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:12.730944 (+ 3125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:12.730969 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:12.734173 (+ 3204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:12.734182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:12.734428 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:12.734471 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 01:07:12.734571 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 01:07:12.734748 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19068,"thread_start_us":18708,"threads_started":1}
I20260619 01:07:12.736555 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bb91bdb1d574a71a7cf8932a3895367" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "467da69d46374637abe0bfa08bd696fb" is_pre_election: true
I20260619 01:07:12.736953 7145 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bb91bdb1d574a71a7cf8932a3895367 in term 1.
I20260619 01:07:12.736897 7341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "467da69d46374637abe0bfa08bd696fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5bb91bdb1d574a71a7cf8932a3895367" is_pre_election: true
I20260619 01:07:12.737273 7341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 467da69d46374637abe0bfa08bd696fb in term 1.
I20260619 01:07:12.737578 7317 leader_election.cc:302] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 467da69d46374637abe0bfa08bd696fb, 5bb91bdb1d574a71a7cf8932a3895367; no voters: 7cf28be23db947b1be7b7bc45f6e54a2
I20260619 01:07:12.737931 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.737453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48044)
0619 01:07:12.737638 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.737734 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 01:07:12.737880 7121 leader_election.cc:302] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 467da69d46374637abe0bfa08bd696fb, 5bb91bdb1d574a71a7cf8932a3895367; no voters:
I20260619 01:07:12.738014 7875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 01:07:12.738130 7875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7cf28be23db947b1be7b7bc45f6e54a2)
I20260619 01:07:12.738157 7874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 01:07:12.738233 7875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:07:12.738270 7874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7cf28be23db947b1be7b7bc45f6e54a2)
W20260619 01:07:12.738263 7119 leader_election.cc:334] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437): Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
I20260619 01:07:12.738415 7874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:07:12.741570 7874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } }
I20260619 01:07:12.741570 7875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } }
I20260619 01:07:12.742413 7875 leader_election.cc:288] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [CANDIDATE]: Term 2 election: Requested vote from peers 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437), 467da69d46374637abe0bfa08bd696fb (127.0.97.61:44201)
I20260619 01:07:12.742440 7874 leader_election.cc:288] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 2 election: Requested vote from peers 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437), 5bb91bdb1d574a71a7cf8932a3895367 (127.0.97.60:37751)
I20260619 01:07:12.742995 7341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "467da69d46374637abe0bfa08bd696fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5bb91bdb1d574a71a7cf8932a3895367"
I20260619 01:07:12.743129 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bb91bdb1d574a71a7cf8932a3895367" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "467da69d46374637abe0bfa08bd696fb"
I20260619 01:07:12.743323 7341 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 467da69d46374637abe0bfa08bd696fb in current term 2: Already voted for candidate 5bb91bdb1d574a71a7cf8932a3895367 in this term.
I20260619 01:07:12.743458 7145 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bb91bdb1d574a71a7cf8932a3895367 in current term 2: Already voted for candidate 467da69d46374637abe0bfa08bd696fb in this term.
I20260619 01:07:12.743507 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.742873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48046)
0619 01:07:12.743134 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.743282 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260619 01:07:12.743513 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.743099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48054)
0619 01:07:12.743249 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.743347 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 01:07:12.744719 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.744300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48060)
0619 01:07:12.744496 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.744574 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 01:07:12.744760 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:12.744415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48064)
0619 01:07:12.744565 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:12.744639 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 01:07:12.744916 7119 leader_election.cc:334] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437): Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
W20260619 01:07:12.744973 7315 leader_election.cc:334] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437): Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
I20260619 01:07:12.745023 7119 leader_election.cc:302] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 467da69d46374637abe0bfa08bd696fb; no voters: 5bb91bdb1d574a71a7cf8932a3895367, 7cf28be23db947b1be7b7bc45f6e54a2
I20260619 01:07:12.745115 7315 leader_election.cc:302] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bb91bdb1d574a71a7cf8932a3895367; no voters: 467da69d46374637abe0bfa08bd696fb, 7cf28be23db947b1be7b7bc45f6e54a2
I20260619 01:07:12.745412 7874 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 01:07:12.745422 7875 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 01:07:13.184242 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:13.183463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.2:33665)
0619 01:07:13.184000 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:13.184087 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":305,"threads_started":1}
I20260619 01:07:13.187911 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:13.187136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.1:54395)
0619 01:07:13.187647 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:13.187727 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":276,"threads_started":1}
I20260619 01:07:13.191282 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:13.190535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.3:58933)
0619 01:07:13.191038 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:13.191114 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":284,"threads_started":1}
I20260619 01:07:14.186299 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.185482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.2:49515)
0619 01:07:14.186004 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.186091 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":276,"threads_started":1}
I20260619 01:07:14.189579 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.188769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.1:54049)
0619 01:07:14.189246 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.189412 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":254,"threads_started":1}
I20260619 01:07:14.192932 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.192138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.3:44541)
0619 01:07:14.192614 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.192701 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":270,"threads_started":1}
I20260619 01:07:14.418562 7890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:07:14.418820 7890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } }
I20260619 01:07:14.419492 7890 leader_election.cc:288] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437), 5bb91bdb1d574a71a7cf8932a3895367 (127.0.97.60:37751)
I20260619 01:07:14.420125 7341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "467da69d46374637abe0bfa08bd696fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5bb91bdb1d574a71a7cf8932a3895367" is_pre_election: true
I20260619 01:07:14.420384 7341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 467da69d46374637abe0bfa08bd696fb in term 2.
I20260619 01:07:14.420768 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.419713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48328)
0619 01:07:14.420369 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.420467 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":414,"threads_started":1}
I20260619 01:07:14.420892 7121 leader_election.cc:302] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 467da69d46374637abe0bfa08bd696fb, 5bb91bdb1d574a71a7cf8932a3895367; no voters:
I20260619 01:07:14.421305 7890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 01:07:14.421461 7890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:07:14.421588 7890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 2 FOLLOWER]: Advancing to term 3
I20260619 01:07:14.422380 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.421944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48338)
0619 01:07:14.422085 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.422217 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 01:07:14.422559 7119 leader_election.cc:334] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437): Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
I20260619 01:07:14.424162 7890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } }
I20260619 01:07:14.424764 7890 leader_election.cc:288] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 3 election: Requested vote from peers 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437), 5bb91bdb1d574a71a7cf8932a3895367 (127.0.97.60:37751)
I20260619 01:07:14.425388 7341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "467da69d46374637abe0bfa08bd696fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5bb91bdb1d574a71a7cf8932a3895367"
I20260619 01:07:14.425473 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.425009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48350)
0619 01:07:14.425175 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.425258 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 01:07:14.425616 7341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 2 FOLLOWER]: Advancing to term 3
I20260619 01:07:14.426550 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.426227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48362)
0619 01:07:14.426380 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.426447 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 01:07:14.426702 7119 leader_election.cc:334] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437): Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
I20260619 01:07:14.428107 7341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 467da69d46374637abe0bfa08bd696fb in term 3.
I20260619 01:07:14.428619 7121 leader_election.cc:302] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 467da69d46374637abe0bfa08bd696fb, 5bb91bdb1d574a71a7cf8932a3895367; no voters: 7cf28be23db947b1be7b7bc45f6e54a2
I20260619 01:07:14.428937 7890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 3 FOLLOWER]: Leader election won for term 3
I20260619 01:07:14.429456 7890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [term 3 LEADER]: Becoming Leader. State: Replica: 467da69d46374637abe0bfa08bd696fb, State: Running, Role: LEADER
I20260619 01:07:14.429845 7890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } }
I20260619 01:07:14.431738 7893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467da69d46374637abe0bfa08bd696fb. Latest consensus state: current_term: 3 leader_uuid: "467da69d46374637abe0bfa08bd696fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:14.432081 7893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:14.432737 7895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:07:14.436472 7895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:07:14.437446 7895 catalog_manager.cc:1259] Loaded cluster ID: 6ef0b99fb5d24edda85c4a6401c5bfc3
I20260619 01:07:14.437518 7895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:07:14.438776 7895 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:07:14.439529 7895 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: Loaded TSK: 0
I20260619 01:07:14.440328 7895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:07:14.471738 7134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:07:14.479035 7134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:07:14.483330 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.482901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48386)
0619 01:07:14.483095 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.483186 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 01:07:14.483381 7341 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [term 3 FOLLOWER]: Refusing update from remote peer 467da69d46374637abe0bfa08bd696fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 01:07:14.484313 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.483918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48396)
0619 01:07:14.484097 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.484165 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 01:07:14.484171 7893 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 01:07:14.484549 7119 consensus_peers.cc:487] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb -> Peer 7cf28be23db947b1be7b7bc45f6e54a2 (127.0.97.62:40437): Couldn't send request to peer 7cf28be23db947b1be7b7bc45f6e54a2. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 01:07:14.487139 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467da69d46374637abe0bfa08bd696fb. Latest consensus state: current_term: 3 leader_uuid: "467da69d46374637abe0bfa08bd696fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:14.487408 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:07:14.489080 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "467da69d46374637abe0bfa08bd696fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:14.489228 7898 mvcc.cc:205] Tried to move back new op lower bound from 7298380736436908032 to 7298380736227983360. Current Snapshot: MvccSnapshot[applied={T|T < 7298380736436908032}]
I20260619 01:07:14.489439 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:14.489986 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "467da69d46374637abe0bfa08bd696fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7cf28be23db947b1be7b7bc45f6e54a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40437 } } peers { permanent_uuid: "467da69d46374637abe0bfa08bd696fb" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44201 } attrs { promote: false } } peers { permanent_uuid: "5bb91bdb1d574a71a7cf8932a3895367" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37751 } attrs { promote: false } } }
I20260619 01:07:14.490338 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb91bdb1d574a71a7cf8932a3895367 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:07:14.519464 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.499475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44720 (local address 127.0.97.1:45717)
0619 01:07:14.511197 (+ 11722us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:14.511206 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:14.511235 (+ 29us) server_negotiation.cc:407] Connection header received
0619 01:07:14.511334 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:14.511339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:14.511466 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:14.511617 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:14.512716 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.513705 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.517288 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.518490 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.518636 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.518756 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:14.518995 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 01:07:14.519128 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11536,"thread_start_us":11402,"threads_started":1}
I20260619 01:07:14.519536 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.499091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45717 (local address 127.0.0.1:44720)
0619 01:07:14.499831 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.499870 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:14.500011 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:14.511693 (+ 11682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:14.511702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:14.511748 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:14.512452 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:14.512470 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.513836 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.513841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:14.516654 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:14.516669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.518682 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.518687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:14.518815 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.518840 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 01:07:14.518963 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 01:07:14.519109 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":11,"thread_start_us":627,"threads_started":1}
I20260619 01:07:14.520251 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.499753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.0.97.2:42509)
0619 01:07:14.508877 (+ 9124us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:14.508888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:14.513207 (+ 4319us) server_negotiation.cc:407] Connection header received
0619 01:07:14.513408 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:14.513414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:14.513491 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:14.513594 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:14.514882 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.515865 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.517798 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.519036 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.519153 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.519248 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:14.519770 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 01:07:14.519923 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8928,"thread_start_us":8776,"threads_started":1}
I20260619 01:07:14.521080 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.498662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41499 (local address 127.0.0.1:42302)
0619 01:07:14.498827 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.498868 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:14.499011 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:14.513435 (+ 14424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:14.513439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:14.513459 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:14.513774 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:14.513785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.515252 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.515260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:14.517309 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:14.517322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.520364 (+ 3042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.520372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:14.520536 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.520558 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:07:14.520630 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 01:07:14.520764 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":49}
I20260619 01:07:14.521121 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.498802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.0.97.3:41499)
0619 01:07:14.512345 (+ 13543us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:14.512349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:14.512392 (+ 43us) server_negotiation.cc:407] Connection header received
0619 01:07:14.512464 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:14.512468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:14.512539 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:14.513438 (+ 899us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:14.513929 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.515084 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.518907 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.520147 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.520291 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.520392 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:14.520661 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 01:07:14.520802 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13419,"thread_start_us":13309,"threads_started":1}
I20260619 01:07:14.521751 7445 tablet_service.cc:1467] Processing CreateTablet for tablet 31677d7ac9c44bc9b949cd33f8b4b52f (DEFAULT_TABLE table=test-table [id=3a3bbe7f1d1d4eba99b370cbcab2d210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:14.522081 7732 tablet_service.cc:1467] Processing CreateTablet for tablet 31677d7ac9c44bc9b949cd33f8b4b52f (DEFAULT_TABLE table=test-table [id=3a3bbe7f1d1d4eba99b370cbcab2d210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:14.522254 7445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31677d7ac9c44bc9b949cd33f8b4b52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:14.522507 7732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31677d7ac9c44bc9b949cd33f8b4b52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:14.521135 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.499586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42509 (local address 127.0.0.1:49944)
0619 01:07:14.513077 (+ 13491us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.513100 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:14.513254 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:14.513793 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:14.513797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:14.513820 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:14.514643 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:14.514654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.516028 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.516032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:14.517639 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:14.517651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.519404 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.519410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:14.519589 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.519621 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:07:14.520530 (+ 909us) client_negotiation.cc:240] Negotiation successful
0619 01:07:14.520677 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13374,"mutex_wait_us":32,"thread_start_us":13249,"threads_started":1}
I20260619 01:07:14.527242 7908 tablet_bootstrap.cc:492] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a: Bootstrap starting.
I20260619 01:07:14.527841 7909 tablet_bootstrap.cc:492] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6: Bootstrap starting.
I20260619 01:07:14.528347 7591 tablet_service.cc:1467] Processing CreateTablet for tablet 31677d7ac9c44bc9b949cd33f8b4b52f (DEFAULT_TABLE table=test-table [id=3a3bbe7f1d1d4eba99b370cbcab2d210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:14.528831 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31677d7ac9c44bc9b949cd33f8b4b52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:14.529692 7909 tablet_bootstrap.cc:654] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:14.529695 7908 tablet_bootstrap.cc:654] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:14.531736 7909 tablet_bootstrap.cc:492] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6: No bootstrap required, opened a new log
I20260619 01:07:14.531910 7909 ts_tablet_manager.cc:1397] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 01:07:14.532723 7909 raft_consensus.cc:348] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:14.533000 7908 tablet_bootstrap.cc:492] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a: No bootstrap required, opened a new log
I20260619 01:07:14.532997 7909 raft_consensus.cc:374] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:14.533093 7909 raft_consensus.cc:729] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d1a013d240424c8a0855353feed2f6, State: Initialized, Role: FOLLOWER
I20260619 01:07:14.533187 7908 ts_tablet_manager.cc:1397] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 01:07:14.533413 7909 consensus_queue.cc:260] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:14.534087 7908 raft_consensus.cc:348] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:14.534289 7909 ts_tablet_manager.cc:1428] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:07:14.534392 7908 raft_consensus.cc:374] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:14.534505 7908 raft_consensus.cc:729] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4233d57aad4aa991a36d8abcd4913a, State: Initialized, Role: FOLLOWER
I20260619 01:07:14.534762 7908 consensus_queue.cc:260] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:14.535467 7913 tablet_bootstrap.cc:492] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7: Bootstrap starting.
I20260619 01:07:14.536188 7908 ts_tablet_manager.cc:1428] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:14.536485 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.535472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.3:45313)
0619 01:07:14.536210 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.536317 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620}
I20260619 01:07:14.537514 7913 tablet_bootstrap.cc:654] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:14.539141 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.538671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.1:40913)
0619 01:07:14.538838 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.538968 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 01:07:14.540287 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.539807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.1:32803)
0619 01:07:14.540014 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.540094 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 01:07:14.540303 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.539799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.3:35809)
0619 01:07:14.540023 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.540123 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260619 01:07:14.542101 7512 heartbeater.cc:503] Master 127.0.97.61:44201 was elected leader, sending a full tablet report...
I20260619 01:07:14.542261 7799 heartbeater.cc:503] Master 127.0.97.61:44201 was elected leader, sending a full tablet report...
I20260619 01:07:14.543785 7913 tablet_bootstrap.cc:492] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7: No bootstrap required, opened a new log
I20260619 01:07:14.544087 7913 ts_tablet_manager.cc:1397] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260619 01:07:14.544797 7913 raft_consensus.cc:348] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:14.544993 7913 raft_consensus.cc:374] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:14.545053 7913 raft_consensus.cc:729] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e5e05e6f2a4c08a8454513dbd696a7, State: Initialized, Role: FOLLOWER
I20260619 01:07:14.545217 7913 consensus_queue.cc:260] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:14.546041 7913 ts_tablet_manager.cc:1428] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:07:14.546911 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.546457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.2:33013)
0619 01:07:14.546630 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.546749 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 01:07:14.548097 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.547613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.2:58059)
0619 01:07:14.547791 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.547899 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 01:07:14.548470 7658 heartbeater.cc:503] Master 127.0.97.61:44201 was elected leader, sending a full tablet report...
I20260619 01:07:14.690100 7914 raft_consensus.cc:482] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:07:14.690342 7914 raft_consensus.cc:504] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:14.692227 7914 leader_election.cc:288] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d1a013d240424c8a0855353feed2f6 (127.0.97.3:41499), a6e5e05e6f2a4c08a8454513dbd696a7 (127.0.97.2:42509)
I20260619 01:07:14.700703 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.692646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49639 (local address 127.0.97.3:41499)
0619 01:07:14.692858 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:14.692864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:14.693009 (+ 145us) server_negotiation.cc:407] Connection header received
0619 01:07:14.693350 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:14.693358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:14.693443 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:14.693560 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:14.694059 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.695199 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.698081 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.699373 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.699542 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.699660 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:14.700275 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 01:07:14.700420 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:07:14.700708 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.692745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36153 (local address 127.0.97.2:42509)
0619 01:07:14.693001 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:14.693007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:14.693618 (+ 611us) server_negotiation.cc:407] Connection header received
0619 01:07:14.693839 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:14.693844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:14.693902 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:14.694012 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:14.694478 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.695432 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.698080 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.699373 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.699542 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.699660 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:14.700275 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 01:07:14.700436 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 01:07:14.700856 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.692480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41499 (local address 127.0.97.1:49639)
0619 01:07:14.692876 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.692904 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:14.693090 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:14.693579 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:14.693584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:14.693608 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:14.693899 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:14.693908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.695343 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.695346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:14.697847 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:14.697870 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.699744 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.699749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:14.700052 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.700123 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 01:07:14.700229 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 01:07:14.700428 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":148,"spinlock_wait_cycles":11520}
I20260619 01:07:14.700920 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.692480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42509 (local address 127.0.97.1:36153)
0619 01:07:14.693510 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.693525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:14.693681 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:14.694059 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:14.694064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:14.694096 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:14.694336 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:14.694342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.695555 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.695561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:14.697851 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:14.697870 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.699730 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.699738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:14.700024 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.700116 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 01:07:14.700229 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 01:07:14.700428 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":348,"spinlock_wait_cycles":768,"thread_start_us":530,"threads_started":1}
I20260619 01:07:14.701932 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31677d7ac9c44bc9b949cd33f8b4b52f" candidate_uuid: "bd4233d57aad4aa991a36d8abcd4913a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d1a013d240424c8a0855353feed2f6" is_pre_election: true
I20260619 01:07:14.701949 7611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31677d7ac9c44bc9b949cd33f8b4b52f" candidate_uuid: "bd4233d57aad4aa991a36d8abcd4913a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" is_pre_election: true
I20260619 01:07:14.702216 7752 raft_consensus.cc:2454] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd4233d57aad4aa991a36d8abcd4913a in term 0.
I20260619 01:07:14.702370 7611 raft_consensus.cc:2454] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd4233d57aad4aa991a36d8abcd4913a in term 0.
I20260619 01:07:14.702889 7398 leader_election.cc:302] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d1a013d240424c8a0855353feed2f6, bd4233d57aad4aa991a36d8abcd4913a; no voters:
I20260619 01:07:14.703329 7914 raft_consensus.cc:2781] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:07:14.703442 7914 raft_consensus.cc:482] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:07:14.703558 7914 raft_consensus.cc:3037] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:14.705796 7914 raft_consensus.cc:504] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:14.706471 7914 leader_election.cc:288] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [CANDIDATE]: Term 1 election: Requested vote from peers 08d1a013d240424c8a0855353feed2f6 (127.0.97.3:41499), a6e5e05e6f2a4c08a8454513dbd696a7 (127.0.97.2:42509)
I20260619 01:07:14.707103 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31677d7ac9c44bc9b949cd33f8b4b52f" candidate_uuid: "bd4233d57aad4aa991a36d8abcd4913a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d1a013d240424c8a0855353feed2f6"
I20260619 01:07:14.707286 7752 raft_consensus.cc:3037] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:14.708019 7611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31677d7ac9c44bc9b949cd33f8b4b52f" candidate_uuid: "bd4233d57aad4aa991a36d8abcd4913a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7"
I20260619 01:07:14.708248 7611 raft_consensus.cc:3037] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:14.710426 7752 raft_consensus.cc:2454] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd4233d57aad4aa991a36d8abcd4913a in term 1.
I20260619 01:07:14.710934 7398 leader_election.cc:302] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d1a013d240424c8a0855353feed2f6, bd4233d57aad4aa991a36d8abcd4913a; no voters:
I20260619 01:07:14.711299 7914 raft_consensus.cc:2781] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:14.711431 7611 raft_consensus.cc:2454] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd4233d57aad4aa991a36d8abcd4913a in term 1.
I20260619 01:07:14.711935 7914 raft_consensus.cc:686] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 1 LEADER]: Becoming Leader. State: Replica: bd4233d57aad4aa991a36d8abcd4913a, State: Running, Role: LEADER
I20260619 01:07:14.712616 7914 consensus_queue.cc:237] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } }
I20260619 01:07:14.713025 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.712509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.1:41667)
0619 01:07:14.712744 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.712852 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 01:07:14.716322 7135 catalog_manager.cc:5495] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a reported cstate change: term changed from 0 to 1, leader changed from <none> to bd4233d57aad4aa991a36d8abcd4913a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "bd4233d57aad4aa991a36d8abcd4913a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd4233d57aad4aa991a36d8abcd4913a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 } health_report { overall_health: UNKNOWN } } }
I20260619 01:07:14.750599 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.745084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.0.97.1:45717)
0619 01:07:14.745354 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:14.745360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:14.745381 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:07:14.745550 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:14.745554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:14.745635 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:14.745801 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:07:14.746184 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.746892 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.748273 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.748705 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.748849 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.749395 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:07:14.749977 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:07:14.750150 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:14.750231 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 01:07:14.750316 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:07:14.755102 7752 raft_consensus.cc:1261] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6 [term 1 FOLLOWER]: Refusing update from remote peer bd4233d57aad4aa991a36d8abcd4913a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:14.755359 7611 raft_consensus.cc:1261] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7 [term 1 FOLLOWER]: Refusing update from remote peer bd4233d57aad4aa991a36d8abcd4913a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:14.755934 7918 consensus_queue.cc:1035] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d1a013d240424c8a0855353feed2f6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:14.756313 7914 consensus_queue.cc:1035] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e5e05e6f2a4c08a8454513dbd696a7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:14.760166 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.758989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.3:38815)
0619 01:07:14.759790 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.759932 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641}
I20260619 01:07:14.761265 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.760686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.2:51469)
0619 01:07:14.760926 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.761062 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 01:07:14.765232 7921 mvcc.cc:205] Tried to move back new op lower bound from 7298380737548931072 to 7298380737386872832. Current Snapshot: MvccSnapshot[applied={T|T < 7298380737548931072}]
I20260619 01:07:14.766855 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.766299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.1:39973)
0619 01:07:14.766555 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.766669 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260619 01:07:14.768430 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.767905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.2:46487)
0619 01:07:14.768134 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.768226 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 01:07:14.769892 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.769395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.1:47423)
0619 01:07:14.769614 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.769712 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 01:07:14.770114 7922 mvcc.cc:205] Tried to move back new op lower bound from 7298380737548931072 to 7298380737386872832. Current Snapshot: MvccSnapshot[applied={T|T < 7298380737548931072}]
I20260619 01:07:14.772419 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.771655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.97.3:38149)
0619 01:07:14.772069 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:14.772193 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260619 01:07:14.785784 7134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:14.786082 7134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:14.792428 7134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 467da69d46374637abe0bfa08bd696fb: Sending DeleteTablet for 3 replicas of tablet 31677d7ac9c44bc9b949cd33f8b4b52f
I20260619 01:07:14.793578 7732 tablet_service.cc:1514] Processing DeleteTablet for tablet 31677d7ac9c44bc9b949cd33f8b4b52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:14 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:42302
I20260619 01:07:14.793705 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet 31677d7ac9c44bc9b949cd33f8b4b52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:14 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:49944
I20260619 01:07:14.794148 7445 tablet_service.cc:1514] Processing DeleteTablet for tablet 31677d7ac9c44bc9b949cd33f8b4b52f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:14 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44720
I20260619 01:07:14.795775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 7312
I20260619 01:07:14.798388 7931 tablet_replica.cc:330] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7: stopping tablet replica
I20260619 01:07:14.798673 7931 raft_consensus.cc:2229] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:14.799098 7931 raft_consensus.cc:2258] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:14.800765 7931 ts_tablet_manager.cc:1905] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:14.805923 7932 tablet_replica.cc:330] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a: stopping tablet replica
I20260619 01:07:14.806533 7932 raft_consensus.cc:2229] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:07:14.807569 7932 raft_consensus.cc:2258] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:14.811295 7932 ts_tablet_manager.cc:1905] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:14.815382 7930 tablet_replica.cc:330] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6: stopping tablet replica
I20260619 01:07:14.815629 7930 raft_consensus.cc:2229] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:14.815968 7930 raft_consensus.cc:2258] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:14.821900 7932 ts_tablet_manager.cc:1918] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:14.822211 7932 log.cc:1196] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/31677d7ac9c44bc9b949cd33f8b4b52f
I20260619 01:07:14.822499 7930 ts_tablet_manager.cc:1905] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:14.822918 7932 ts_tablet_manager.cc:1939] T 31677d7ac9c44bc9b949cd33f8b4b52f P bd4233d57aad4aa991a36d8abcd4913a: Deleting consensus metadata
I20260619 01:07:14.824339 7122 catalog_manager.cc:4841] TS bd4233d57aad4aa991a36d8abcd4913a (127.0.97.1:45717): tablet 31677d7ac9c44bc9b949cd33f8b4b52f (table test-table [id=3a3bbe7f1d1d4eba99b370cbcab2d210]) successfully deleted
I20260619 01:07:14.827397 7931 ts_tablet_manager.cc:1918] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:14.827557 7931 log.cc:1196] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/31677d7ac9c44bc9b949cd33f8b4b52f
I20260619 01:07:14.828050 7931 ts_tablet_manager.cc:1939] T 31677d7ac9c44bc9b949cd33f8b4b52f P a6e5e05e6f2a4c08a8454513dbd696a7: Deleting consensus metadata
I20260619 01:07:14.829447 7120 catalog_manager.cc:4841] TS a6e5e05e6f2a4c08a8454513dbd696a7 (127.0.97.2:42509): tablet 31677d7ac9c44bc9b949cd33f8b4b52f (table test-table [id=3a3bbe7f1d1d4eba99b370cbcab2d210]) successfully deleted
I20260619 01:07:14.829493 7930 ts_tablet_manager.cc:1918] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:14.829627 7930 log.cc:1196] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/31677d7ac9c44bc9b949cd33f8b4b52f
I20260619 01:07:14.830058 7930 ts_tablet_manager.cc:1939] T 31677d7ac9c44bc9b949cd33f8b4b52f P 08d1a013d240424c8a0855353feed2f6: Deleting consensus metadata
I20260619 01:07:14.831344 7119 catalog_manager.cc:4841] TS 08d1a013d240424c8a0855353feed2f6 (127.0.97.3:41499): tablet 31677d7ac9c44bc9b949cd33f8b4b52f (table test-table [id=3a3bbe7f1d1d4eba99b370cbcab2d210]) successfully deleted
I20260619 01:07:14.838398 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.797284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.0.97.1:45717)
0619 01:07:14.797504 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:14.797513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:14.808414 (+ 10901us) server_negotiation.cc:407] Connection header received
0619 01:07:14.811996 (+ 3582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:14.812007 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:14.812128 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:14.812219 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:14.821602 (+ 9383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.822910 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.824841 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.825533 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.825662 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.830564 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:14.830588 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:14.830613 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:14.830674 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:14.834900 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:14.835963 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:14.835972 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:14.835975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:14.836146 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:14.836569 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:14.836573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:14.836575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:14.837115 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:14.837524 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:14.837744 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 01:07:14.838015 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 01:07:14.839133 7505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:14.945583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 7312
I20260619 01:07:14.964370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 7522
I20260619 01:07:14.978942 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:14.965170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.0.97.2:42509)
0619 01:07:14.965428 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:14.965435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:14.965645 (+ 210us) server_negotiation.cc:407] Connection header received
0619 01:07:14.965788 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:14.965793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:14.965901 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:14.965997 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:14.967507 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.968364 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.969897 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:14.970323 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:14.970445 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:14.973055 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:14.973068 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:14.973085 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:14.973116 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:14.976228 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:14.976988 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:14.976996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:14.976998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:14.977138 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:14.977542 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:14.977545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:14.977547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:14.977899 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:14.978136 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:14.978376 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 01:07:14.978550 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 01:07:14.979698 7651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:15.078316 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:15.064397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48410)
0619 01:07:15.077893 (+ 13496us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:15.078047 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13318,"thread_start_us":13201,"threads_started":1}
I20260619 01:07:15.080119 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:15.079609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40437 (local address 127.0.0.1:48420)
0619 01:07:15.079817 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:15.079941 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:40437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 01:07:15.112920 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 7522
I20260619 01:07:15.145948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 7669
I20260619 01:07:15.163921 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:15.146823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.0.97.3:41499)
0619 01:07:15.147048 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:15.147056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:15.147084 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:07:15.147194 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:15.147203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:15.147300 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:15.147432 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:15.148796 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:15.150245 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:15.151555 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:15.152439 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:15.152613 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:15.155354 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:15.155378 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:15.155405 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:15.155462 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:15.160394 (+ 4932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:15.161262 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:15.161275 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:15.161282 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:15.161520 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:15.161939 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:15.161947 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:15.161952 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:15.162490 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:15.162919 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:15.163098 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 01:07:15.163383 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 01:07:15.164687 7792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:15.259274 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 7669
I20260619 01:07:15.277410 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 6998
I20260619 01:07:15.278584 7165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:15.400456 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 6998
W20260619 01:07:15.423558 7318 connection.cc:535] server connection from 127.0.0.1:48586 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 01:07:15.423870 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 7099
I20260619 01:07:15.424882 7361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:15.540457 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 7099
2026-06-19T01:07:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11873 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:07:15 dist-test-slave-5p8w krb5kdc[7943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:07:15 dist-test-slave-5p8w krb5kdc[7943](info): set up 2 sockets
Jun 19 01:07:15 dist-test-slave-5p8w krb5kdc[7943](info): commencing operation
krb5kdc: starting...
W20260619 01:07:16.654196 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:07:16 dist-test-slave-5p8w krb5kdc[7943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:07:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 01:07:16.889038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36229
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:17.235766 7956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:17.236086 7956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:17.236272 7956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:17.243669 7956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:07:17.243800 7956 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 01:07:17.243889 7956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:17.243989 7956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:17.244074 7956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:07:17.244186 7956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:07:17.254536 7956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36229
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:17.255980 7956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:17.257436 7956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:17.270675 7962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:17.270764 7961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:17.270726 7964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:17.271298 7956 server_base.cc:1034] running on GCE node
I20260619 01:07:17.272365 7956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:17.273972 7956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:17.275228 7956 hybrid_clock.cc:648] HybridClock initialized: now 1781831237275139 us; error 64 us; skew 500 ppm
I20260619 01:07:17.275558 7956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:17 dist-test-slave-5p8w krb5kdc[7943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:17.279052 7956 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 01:07:17.280670 7956 webserver.cc:460] Webserver started at http://127.0.97.62:46051/ using document root <none> and password file <none>
I20260619 01:07:17.281533 7956 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:17.281628 7956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:17.281912 7956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:17.284924 7956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "be08ffb30015417985bb14e2a4360beb"
format_stamp: "Formatted at 2026-06-19 01:07:17 on dist-test-slave-5p8w"
server_key: "f271839dd625776dec813c26f2e0839a"
server_key_iv: "ad1cbae31265176ff5382fb692130f15"
server_key_version: "encryptionkey@0"
I20260619 01:07:17.285820 7956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "be08ffb30015417985bb14e2a4360beb"
format_stamp: "Formatted at 2026-06-19 01:07:17 on dist-test-slave-5p8w"
server_key: "f271839dd625776dec813c26f2e0839a"
server_key_iv: "ad1cbae31265176ff5382fb692130f15"
server_key_version: "encryptionkey@0"
I20260619 01:07:17.291679 7956 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 01:07:17.295658 7971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:17.297255 7956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:07:17.297497 7956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "be08ffb30015417985bb14e2a4360beb"
format_stamp: "Formatted at 2026-06-19 01:07:17 on dist-test-slave-5p8w"
server_key: "f271839dd625776dec813c26f2e0839a"
server_key_iv: "ad1cbae31265176ff5382fb692130f15"
server_key_version: "encryptionkey@0"
I20260619 01:07:17.297703 7956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:17.320873 7956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:17.325490 7956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:17.325801 7956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:17.348520 7956 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36229
I20260619 01:07:17.348526 8032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36229 every 8 connection(s)
I20260619 01:07:17.350132 7956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:07:17.352568 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 7956
I20260619 01:07:17.352813 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:07:17.353495 388 external_mini_cluster.cc:1428] Setting key d85ba9b7fc0f5d47c6ab160cd8caa9b0
I20260619 01:07:17.354808 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:07:17 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:17.371330 8033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb: Bootstrap starting.
I20260619 01:07:17.374737 8033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:17.375873 8033 log.cc:824] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:17.378739 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:17.355639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.0.97.62:36229)
0619 01:07:17.356545 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:17.356558 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:17.356601 (+ 43us) server_negotiation.cc:407] Connection header received
0619 01:07:17.358340 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:17.358367 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:17.358864 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:17.359261 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:17.361463 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:17.363033 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:17.365105 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:17.365956 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:17.366106 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:17.370977 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:17.370994 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:17.371013 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:17.371057 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:17.374292 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:17.375120 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:17.375128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:17.375136 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:17.375316 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:17.375742 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:17.375747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:17.375750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:17.376398 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:17.376720 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:17.377258 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 01:07:17.377726 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":399,"threads_started":1}
I20260619 01:07:17.379078 8033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb: No bootstrap required, opened a new log
I20260619 01:07:17.387367 8033 raft_consensus.cc:348] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08ffb30015417985bb14e2a4360beb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } }
I20260619 01:07:17.387825 8033 raft_consensus.cc:374] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:17.387946 8033 raft_consensus.cc:729] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be08ffb30015417985bb14e2a4360beb, State: Initialized, Role: FOLLOWER
I20260619 01:07:17.388612 8033 consensus_queue.cc:260] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08ffb30015417985bb14e2a4360beb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } }
I20260619 01:07:17.388870 8033 raft_consensus.cc:388] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:07:17.389021 8033 raft_consensus.cc:482] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:07:17.389211 8033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:17.391602 8033 raft_consensus.cc:504] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08ffb30015417985bb14e2a4360beb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } }
I20260619 01:07:17.392187 8033 leader_election.cc:302] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be08ffb30015417985bb14e2a4360beb; no voters:
I20260619 01:07:17.392972 8033 leader_election.cc:288] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:07:17.393119 8038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:17.393895 8038 raft_consensus.cc:686] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [term 1 LEADER]: Becoming Leader. State: Replica: be08ffb30015417985bb14e2a4360beb, State: Running, Role: LEADER
I20260619 01:07:17.394447 8038 consensus_queue.cc:237] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08ffb30015417985bb14e2a4360beb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } }
I20260619 01:07:17.395267 8033 sys_catalog.cc:564] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:07:17.397709 8039 sys_catalog.cc:455] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be08ffb30015417985bb14e2a4360beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08ffb30015417985bb14e2a4360beb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } } }
I20260619 01:07:17.397590 8040 sys_catalog.cc:455] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [sys.catalog]: SysCatalogTable state changed. Reason: New leader be08ffb30015417985bb14e2a4360beb. Latest consensus state: current_term: 1 leader_uuid: "be08ffb30015417985bb14e2a4360beb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be08ffb30015417985bb14e2a4360beb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36229 } } }
I20260619 01:07:17.398243 8040 sys_catalog.cc:458] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:17.398902 8039 sys_catalog.cc:458] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:17.399861 8045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:07:17.405453 8045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:07:17.416422 8045 catalog_manager.cc:1347] Generated new cluster ID: 1594b298efab46ac9f0c1163c3b4c40e
I20260619 01:07:17.416565 8045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:07:17.426309 8045 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:07:17.428443 8045 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:07:17.435757 8045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb: Generated new TSK 0
I20260619 01:07:17.437021 8045 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 01:07:17.505780 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36229
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:17.853333 8059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:17.853700 8059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:17.853873 8059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:17.861204 8059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:17.861428 8059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:17.861589 8059 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:07:17.861739 8059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:07:17.871876 8059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:17.873329 8059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:17.874768 8059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:17.889050 8064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:17.888978 8065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:17.889016 8067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:17.889700 8059 server_base.cc:1034] running on GCE node
I20260619 01:07:17.890492 8059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:17.891456 8059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:17.892685 8059 hybrid_clock.cc:648] HybridClock initialized: now 1781831237892620 us; error 51 us; skew 500 ppm
I20260619 01:07:17.893018 8059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:17 dist-test-slave-5p8w krb5kdc[7943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:17.896636 8059 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 01:07:17.898275 8059 webserver.cc:460] Webserver started at http://127.0.97.1:36813/ using document root <none> and password file <none>
I20260619 01:07:17.899055 8059 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:17.899150 8059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:17.899470 8059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:17.902745 8059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "5f711ef0abb8448eba326bc73972e23a"
format_stamp: "Formatted at 2026-06-19 01:07:17 on dist-test-slave-5p8w"
server_key: "edba4ca97b8f05d8b8c104bb378ce719"
server_key_iv: "d1d60e0faa9b060246e1623e21838ab8"
server_key_version: "encryptionkey@0"
I20260619 01:07:17.903661 8059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5f711ef0abb8448eba326bc73972e23a"
format_stamp: "Formatted at 2026-06-19 01:07:17 on dist-test-slave-5p8w"
server_key: "edba4ca97b8f05d8b8c104bb378ce719"
server_key_iv: "d1d60e0faa9b060246e1623e21838ab8"
server_key_version: "encryptionkey@0"
I20260619 01:07:17.909397 8059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 01:07:17.913239 8074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:17.914950 8059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 01:07:17.915153 8059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "5f711ef0abb8448eba326bc73972e23a"
format_stamp: "Formatted at 2026-06-19 01:07:17 on dist-test-slave-5p8w"
server_key: "edba4ca97b8f05d8b8c104bb378ce719"
server_key_iv: "d1d60e0faa9b060246e1623e21838ab8"
server_key_version: "encryptionkey@0"
I20260619 01:07:17.915396 8059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:17.939157 8059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:17.943711 8059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:17.944026 8059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:17.947072 8059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:17.947188 8059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:17.947347 8059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:17.947423 8059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:07:17 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:17.972365 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:17.947713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50683 (local address 127.0.97.62:36229)
0619 01:07:17.948291 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:17.948297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:17.949662 (+ 1365us) server_negotiation.cc:407] Connection header received
0619 01:07:17.951757 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:17.951765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:17.951884 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:17.952014 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:17.953825 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:17.954900 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:17.957801 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:17.958613 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:17.958809 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:17.963714 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:17.963725 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:17.963733 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:17.963796 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:17.967926 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:17.968962 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:17.968971 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:17.968975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:17.969149 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:17.969883 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:17.969887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:17.969889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:17.970315 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:17.970486 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:17.971819 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0619 01:07:17.972058 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":273,"threads_started":1}
I20260619 01:07:17.974547 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:17.948556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36229 (local address 127.0.97.1:50683)
0619 01:07:17.949454 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:17.949509 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:17.951451 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:17.952320 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:17.952331 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:17.952873 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:17.953613 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:17.953638 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:17.955105 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:17.955117 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:17.957570 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:17.957595 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:17.958944 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:17.958957 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:17.959257 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:17.960143 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:17.960206 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:17.963355 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:17.968183 (+ 4828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:17.968194 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:17.968239 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:17.968813 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:17.969283 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:17.969287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:17.969337 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:17.969734 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:17.970489 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:17.970503 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:17.971143 (+ 640us) client_negotiation.cc:769] Sending connection context
0619 01:07:17.971570 (+ 427us) client_negotiation.cc:240] Negotiation successful
0619 01:07:17.972332 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":378,"threads_started":1}
I20260619 01:07:18.001786 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:17.981666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36229 (local address 127.0.97.1:39821)
0619 01:07:17.982768 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:17.982787 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:17.982953 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:17.983437 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:17.983446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:17.984243 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:17.984960 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:17.984985 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:17.986770 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:17.986778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:17.989033 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:17.989065 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:17.990276 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:17.990288 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:17.990471 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:17.991249 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:17.991279 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:17.993009 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:17.997889 (+ 4880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:17.997897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:17.997904 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:17.998780 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:17.999361 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:17.999364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:17.999367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:17.999509 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:18.000271 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:18.000278 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:18.000493 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 01:07:18.000715 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 01:07:18.001124 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":743,"threads_started":1}
I20260619 01:07:18.002101 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:17.981716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39821 (local address 127.0.97.62:36229)
0619 01:07:17.981891 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:17.981895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:17.982916 (+ 1021us) server_negotiation.cc:407] Connection header received
0619 01:07:17.983108 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:17.983117 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:17.983226 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:17.983352 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:17.985213 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:17.986542 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:17.989382 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:17.990014 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:17.990190 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:17.993413 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:17.993423 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:17.993431 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:17.993486 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:17.997642 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:17.998974 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:17.998982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:17.998986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:17.999151 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:17.999658 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:17.999666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:17.999670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:18.000122 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:18.000302 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:18.001489 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0619 01:07:18.001751 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:07:18.008841 8059 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38745
I20260619 01:07:18.008862 8194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38745 every 8 connection(s)
I20260619 01:07:18.010408 8059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:07:18.011323 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 8059
I20260619 01:07:18.011547 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:07:18.012233 388 external_mini_cluster.cc:1428] Setting key c790668351a52ff292eb2e911da6cd33
I20260619 01:07:18.012743 8195 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36229
I20260619 01:07:18.013011 8195 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:18.013885 8195 heartbeater.cc:511] Master 127.0.97.62:36229 requested a full tablet report, sending...
I20260619 01:07:18.015990 7988 ts_manager.cc:194] Registered new tserver with Master: 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745)
I20260619 01:07:18.017645 7988 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50683
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 01:07:18.061224 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36229
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:18.404326 8199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:18.404706 8199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:18.404878 8199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:18.413131 8199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:18.413350 8199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:18.413509 8199 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:07:18.413655 8199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:07:18.425838 8199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:18.427292 8199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:18.428833 8199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:18.443956 8204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:18.443956 8205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:18.443960 8207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:18.444849 8199 server_base.cc:1034] running on GCE node
I20260619 01:07:18.445624 8199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:18.446727 8199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:18.447978 8199 hybrid_clock.cc:648] HybridClock initialized: now 1781831238447891 us; error 51 us; skew 500 ppm
I20260619 01:07:18.448343 8199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:18 dist-test-slave-5p8w krb5kdc[7943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831238, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:18.451850 8199 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 01:07:18.453645 8199 webserver.cc:460] Webserver started at http://127.0.97.2:41223/ using document root <none> and password file <none>
I20260619 01:07:18.454465 8199 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:18.454564 8199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:18.454916 8199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:18.458213 8199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01"
format_stamp: "Formatted at 2026-06-19 01:07:18 on dist-test-slave-5p8w"
server_key: "a9fced1fdae4d8c87a32fe9cec7f0ca6"
server_key_iv: "55003f14092c534881107b5b6a8e832a"
server_key_version: "encryptionkey@0"
I20260619 01:07:18.459123 8199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01"
format_stamp: "Formatted at 2026-06-19 01:07:18 on dist-test-slave-5p8w"
server_key: "a9fced1fdae4d8c87a32fe9cec7f0ca6"
server_key_iv: "55003f14092c534881107b5b6a8e832a"
server_key_version: "encryptionkey@0"
I20260619 01:07:18.464963 8199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 01:07:18.469023 8214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:18.470759 8199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:18.471000 8199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01"
format_stamp: "Formatted at 2026-06-19 01:07:18 on dist-test-slave-5p8w"
server_key: "a9fced1fdae4d8c87a32fe9cec7f0ca6"
server_key_iv: "55003f14092c534881107b5b6a8e832a"
server_key_version: "encryptionkey@0"
I20260619 01:07:18.471278 8199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:18.494854 8199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:18.500062 8199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:18.500468 8199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:18.503925 8199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:18.504050 8199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:18.504205 8199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:18.504315 8199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:07:18 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831238, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:18.533828 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:18.504417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47195 (local address 127.0.97.62:36229)
0619 01:07:18.505268 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:18.505273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:18.506386 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 01:07:18.508843 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:18.508849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:18.508955 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:18.509090 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:18.511852 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:18.513517 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:18.516655 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:18.517461 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:18.517637 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:18.523659 (+ 6022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:18.523671 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:18.523679 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:18.523743 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:18.528654 (+ 4911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:18.530120 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:18.530129 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:18.530133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:18.530360 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:18.531209 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:18.531219 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:18.531225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:18.531686 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:18.531864 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:18.533083 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0619 01:07:18.533416 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":650,"threads_started":1}
I20260619 01:07:18.535068 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:18.505258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36229 (local address 127.0.97.2:47195)
0619 01:07:18.506183 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:18.506240 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:18.508482 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:18.509498 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:18.509515 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:18.510496 (+ 981us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:18.511524 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:18.511565 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:18.513766 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:18.513777 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:18.516389 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:18.516419 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:18.517854 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:18.517867 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:18.518241 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:18.519436 (+ 1195us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:18.519490 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:18.523296 (+ 3806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:18.528933 (+ 5637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:18.528942 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:18.528961 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:18.529845 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:18.530542 (+ 697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:18.530551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:18.530558 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:18.531000 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:18.531876 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:18.531888 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:18.532516 (+ 628us) client_negotiation.cc:769] Sending connection context
0619 01:07:18.532966 (+ 450us) client_negotiation.cc:240] Negotiation successful
0619 01:07:18.533669 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":362,"threads_started":1}
I20260619 01:07:18.560817 8199 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38587
I20260619 01:07:18.560835 8334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38587 every 8 connection(s)
I20260619 01:07:18.562036 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:18.543887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57149 (local address 127.0.97.62:36229)
0619 01:07:18.544130 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:18.544139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:18.545239 (+ 1100us) server_negotiation.cc:407] Connection header received
0619 01:07:18.545550 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:18.545560 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:18.545677 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:18.545797 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:18.547283 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:18.548433 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:18.550594 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:18.551059 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:18.551221 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:18.554319 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:18.554332 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:18.554340 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:18.554400 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:18.558735 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:18.559657 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:18.559668 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:18.559671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:18.559893 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:18.560357 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:18.560364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:18.560367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:18.560706 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:18.560873 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:18.561423 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 01:07:18.561679 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 01:07:18.562150 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:18.544306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36229 (local address 127.0.97.2:57149)
0619 01:07:18.545072 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:18.545101 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:18.545360 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:18.545832 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:18.545837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:18.546468 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:18.546995 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:18.547010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:18.548671 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:18.548677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:18.550396 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:18.550414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:18.551298 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:18.551306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:18.551485 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:18.552339 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:18.552377 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:18.553963 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:18.558991 (+ 5028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:18.558996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:18.559000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:18.559459 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:18.560064 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:18.560069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:18.560072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:18.560220 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:18.560842 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:18.560847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:18.561097 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 01:07:18.561270 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 01:07:18.561701 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":362,"threads_started":1}
I20260619 01:07:18.562435 8199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:07:18.564884 8335 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36229
I20260619 01:07:18.564962 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 8199
I20260619 01:07:18.565156 8335 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:18.565153 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:07:18.565737 388 external_mini_cluster.cc:1428] Setting key 83d6c735f0cef2e25018d4b6c655268c
I20260619 01:07:18.565977 8335 heartbeater.cc:511] Master 127.0.97.62:36229 requested a full tablet report, sending...
I20260619 01:07:18.567817 7987 ts_manager.cc:194] Registered new tserver with Master: 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587)
I20260619 01:07:18.568748 7987 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47195
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 01:07:18.621126 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36229
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:18.975899 8339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:18.976325 8339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:18.976517 8339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:18.984602 8339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:18.984758 8339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:18.984920 8339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:07:18.985015 8339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:07:18.996091 8339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:18.997633 8339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:18.999094 8339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:19.015231 8345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:19.015264 8347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:19.015990 8344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:19.016505 8339 server_base.cc:1034] running on GCE node
I20260619 01:07:19.017431 8339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:19.018497 8339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:19.019835 8339 hybrid_clock.cc:648] HybridClock initialized: now 1781831239019702 us; error 69 us; skew 500 ppm
I20260619 01:07:19.020215 8339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:07:19.021251 8195 heartbeater.cc:503] Master 127.0.97.62:36229 was elected leader, sending a full tablet report...
Jun 19 01:07:19 dist-test-slave-5p8w krb5kdc[7943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831239, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:19.024242 8339 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 01:07:19.025913 8339 webserver.cc:460] Webserver started at http://127.0.97.3:35277/ using document root <none> and password file <none>
I20260619 01:07:19.026654 8339 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:19.026750 8339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:19.027024 8339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:19.030308 8339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "6f58457c81c74bc79e357da433f98539"
format_stamp: "Formatted at 2026-06-19 01:07:19 on dist-test-slave-5p8w"
server_key: "29483b04a504c2a1f18d8148184398f3"
server_key_iv: "7cdb033ab6e7e37e8a8d94747b6c7698"
server_key_version: "encryptionkey@0"
I20260619 01:07:19.031157 8339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6f58457c81c74bc79e357da433f98539"
format_stamp: "Formatted at 2026-06-19 01:07:19 on dist-test-slave-5p8w"
server_key: "29483b04a504c2a1f18d8148184398f3"
server_key_iv: "7cdb033ab6e7e37e8a8d94747b6c7698"
server_key_version: "encryptionkey@0"
I20260619 01:07:19.036755 8339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 01:07:19.040854 8354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:19.042546 8339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:19.042722 8339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "6f58457c81c74bc79e357da433f98539"
format_stamp: "Formatted at 2026-06-19 01:07:19 on dist-test-slave-5p8w"
server_key: "29483b04a504c2a1f18d8148184398f3"
server_key_iv: "7cdb033ab6e7e37e8a8d94747b6c7698"
server_key_version: "encryptionkey@0"
I20260619 01:07:19.042937 8339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:19.055692 8339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:19.060276 8339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:19.060518 8339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:19.063572 8339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:19.063759 8339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 01:07:19.063939 8339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:19.064078 8339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:07:19 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831239, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:19.092044 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.063864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48681 (local address 127.0.97.62:36229)
0619 01:07:19.064333 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:19.064339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:19.065619 (+ 1280us) server_negotiation.cc:407] Connection header received
0619 01:07:19.068708 (+ 3089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:19.068712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:19.068821 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:19.068962 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:19.071982 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.073042 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.076755 (+ 3713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.077401 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.077540 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.083383 (+ 5843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:19.083394 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:19.083402 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:19.083462 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:19.087432 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:19.088977 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:19.088986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:19.088992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:19.089187 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:19.089891 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:19.089896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:19.089898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:19.090285 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:19.090450 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:19.091386 (+ 936us) server_negotiation.cc:299] Negotiation successful
0619 01:07:19.091632 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":234,"threads_started":1}
I20260619 01:07:19.093101 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.064514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36229 (local address 127.0.97.3:48681)
0619 01:07:19.065401 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:19.065456 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:19.068327 (+ 2871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:19.069278 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:19.069333 (+ 55us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:19.070453 (+ 1120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:19.071645 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.071686 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.073255 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.073264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.076400 (+ 3136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.076428 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.077726 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.077738 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.077955 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.078973 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:19.079007 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:19.082983 (+ 3976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:19.087692 (+ 4709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:19.087721 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:19.087747 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:19.088758 (+ 1011us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:19.089381 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:19.089389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:19.089394 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:19.089747 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:19.090537 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:19.090544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:19.090956 (+ 412us) client_negotiation.cc:769] Sending connection context
0619 01:07:19.091261 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 01:07:19.091875 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":339,"threads_started":1}
I20260619 01:07:19.117827 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.100512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55345 (local address 127.0.97.62:36229)
0619 01:07:19.100704 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:19.100709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:19.101357 (+ 648us) server_negotiation.cc:407] Connection header received
0619 01:07:19.101630 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:19.101639 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:19.101740 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:19.101867 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:19.103661 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.104527 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.106847 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.107411 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.107564 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.110766 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:19.110776 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:19.110782 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:19.110832 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:19.113769 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:19.114892 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:19.114900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:19.114905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:19.115043 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:19.115538 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:19.115543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:19.115545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:19.115843 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:19.116001 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:19.117427 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0619 01:07:19.117594 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 01:07:19.118196 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.100521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36229 (local address 127.0.97.3:55345)
0619 01:07:19.101237 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:19.101254 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:19.101449 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:19.102118 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:19.102125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:19.102932 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:19.103419 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.103437 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.104684 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.104692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.106673 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.106692 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.107686 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.107695 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.107956 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.108785 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:19.108815 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:19.110451 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:19.113921 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:19.113927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:19.113936 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:19.114738 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:19.115159 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:19.115164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:19.115179 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:19.115383 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:19.116065 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:19.116073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:19.116275 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 01:07:19.116454 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 01:07:19.116826 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":384,"threads_started":1}
I20260619 01:07:19.119810 8339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42399
I20260619 01:07:19.119853 8474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42399 every 8 connection(s)
I20260619 01:07:19.121568 8339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:07:19.123885 8475 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36229
I20260619 01:07:19.124115 8475 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:19.124663 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 8339
I20260619 01:07:19.124751 8475 heartbeater.cc:511] Master 127.0.97.62:36229 requested a full tablet report, sending...
I20260619 01:07:19.124883 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:07:19.125383 388 external_mini_cluster.cc:1428] Setting key 0362112e8f2ee88bdba7ab623269b2d9
I20260619 01:07:19.126215 7987 ts_manager.cc:194] Registered new tserver with Master: 6f58457c81c74bc79e357da433f98539 (127.0.97.3:42399)
I20260619 01:07:19.127343 7987 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48681
I20260619 01:07:19.128474 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 01:07:19 dist-test-slave-5p8w krb5kdc[7943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831239, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 01:07:19 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:19 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 01:07:19.187249 8362 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:19.172159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.0.97.62:36229)
0619 01:07:19.172355 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:19.172360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:19.172709 (+ 349us) server_negotiation.cc:407] Connection header received
0619 01:07:19.172845 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:19.172849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:19.172920 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:19.173029 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:19.174429 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.175291 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.176640 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.177036 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.177170 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.186986 (+ 9816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37100: BlockingRecv error: recv got EOF from 127.0.0.1:37100 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 01:07:19 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831239, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:19.204962 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.190609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.0.97.62:36229)
0619 01:07:19.190801 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:19.190806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:19.191144 (+ 338us) server_negotiation.cc:407] Connection header received
0619 01:07:19.191299 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:19.191303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:19.191361 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:19.191446 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:19.192715 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.193584 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.195056 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.195690 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.195800 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.199756 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:19.199762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:19.199765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:19.199805 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:19.202702 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:19.203387 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:19.203391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:19.203394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:19.203542 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:19.203863 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:19.203867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:19.203869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:19.204135 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:19.204344 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:19.204564 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 01:07:19.204732 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:07:19.209225 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:07:19.216287 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:07:19.240813 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.229826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38587 (local address 127.0.0.1:45026)
0619 01:07:19.230763 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:19.230803 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:19.231001 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:19.232395 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:19.232405 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:19.232456 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:19.232994 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.233009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.235072 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.235078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.237799 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.237812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.239758 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.239764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.239961 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.240015 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 01:07:19.240178 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 01:07:19.240408 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":3560,"threads_started":1}
I20260619 01:07:19.241484 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.230022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38745 (local address 127.0.0.1:34888)
0619 01:07:19.232750 (+ 2728us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:19.232774 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:19.232998 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:19.234024 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:19.234028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:19.234050 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:19.234496 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.234511 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.236181 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.236186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.238366 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.238378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.240211 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.240218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.240395 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.240428 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 01:07:19.240518 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 01:07:19.240669 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2550,"mutex_wait_us":1,"thread_start_us":302,"threads_started":1}
I20260619 01:07:19.241559 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.230287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34888 (local address 127.0.97.1:38745)
0619 01:07:19.231937 (+ 1650us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:19.231950 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:19.232887 (+ 937us) server_negotiation.cc:407] Connection header received
0619 01:07:19.233240 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:19.233252 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:19.233596 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:19.233760 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:19.234693 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.236006 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.238853 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.239990 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.240138 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.240352 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:19.240607 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 01:07:19.241228 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":1343,"threads_started":1}
I20260619 01:07:19.242494 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.229988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.0.97.2:38587)
0619 01:07:19.231567 (+ 1579us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:19.231579 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:19.231612 (+ 33us) server_negotiation.cc:407] Connection header received
0619 01:07:19.231719 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:19.231727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:19.232043 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:19.233222 (+ 1179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:19.233477 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.234923 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.237994 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.239541 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.239708 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.239960 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:19.241997 (+ 2037us) server_negotiation.cc:299] Negotiation successful
0619 01:07:19.242198 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410,"thread_start_us":1289,"threads_started":1}
I20260619 01:07:19.242779 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.231794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42399 (local address 127.0.0.1:38078)
0619 01:07:19.233686 (+ 1892us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:19.233703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:19.233890 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:19.234766 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:19.234774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:19.234801 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:19.235175 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.235191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.237396 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.237401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.239615 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.239632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.241994 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.242003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.242191 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.242221 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 01:07:19.242314 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 01:07:19.242439 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1772,"thread_start_us":593,"threads_started":1}
I20260619 01:07:19.243558 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.232385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.0.97.3:42399)
0619 01:07:19.233215 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:19.233224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:19.233790 (+ 566us) server_negotiation.cc:407] Connection header received
0619 01:07:19.234062 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:19.234070 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:19.234290 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:19.234444 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:19.235416 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.237175 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.239803 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.241411 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.241530 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.241793 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:19.242391 (+ 598us) server_negotiation.cc:299] Negotiation successful
0619 01:07:19.242578 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":431,"threads_started":1}
I20260619 01:07:19.245852 8265 tablet_service.cc:1467] Processing CreateTablet for tablet b240c9ade04b45f1ac29319c73c3f84a (DEFAULT_TABLE table=test-table [id=a7c9df74549f4c8691e004f59a247dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:19.245819 8125 tablet_service.cc:1467] Processing CreateTablet for tablet b240c9ade04b45f1ac29319c73c3f84a (DEFAULT_TABLE table=test-table [id=a7c9df74549f4c8691e004f59a247dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:19.247300 8125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b240c9ade04b45f1ac29319c73c3f84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:19.247320 8265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b240c9ade04b45f1ac29319c73c3f84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:19.247056 8405 tablet_service.cc:1467] Processing CreateTablet for tablet b240c9ade04b45f1ac29319c73c3f84a (DEFAULT_TABLE table=test-table [id=a7c9df74549f4c8691e004f59a247dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:19.248514 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b240c9ade04b45f1ac29319c73c3f84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:19.259796 8499 tablet_bootstrap.cc:492] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a: Bootstrap starting.
I20260619 01:07:19.262240 8500 tablet_bootstrap.cc:492] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Bootstrap starting.
I20260619 01:07:19.263298 8501 tablet_bootstrap.cc:492] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: Bootstrap starting.
I20260619 01:07:19.263782 8499 tablet_bootstrap.cc:654] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:19.265720 8499 log.cc:824] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:19.266279 8500 tablet_bootstrap.cc:654] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:19.266844 8501 tablet_bootstrap.cc:654] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:19.267637 8500 log.cc:824] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:19.268018 8501 log.cc:824] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:19.269388 8499 tablet_bootstrap.cc:492] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a: No bootstrap required, opened a new log
I20260619 01:07:19.269662 8499 ts_tablet_manager.cc:1397] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 01:07:19.270812 8501 tablet_bootstrap.cc:492] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: No bootstrap required, opened a new log
I20260619 01:07:19.271075 8501 ts_tablet_manager.cc:1397] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 01:07:19.275895 8500 tablet_bootstrap.cc:492] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: No bootstrap required, opened a new log
I20260619 01:07:19.276240 8500 ts_tablet_manager.cc:1397] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260619 01:07:19.277699 8499 raft_consensus.cc:348] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.278194 8499 raft_consensus.cc:374] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:19.278333 8499 raft_consensus.cc:729] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f711ef0abb8448eba326bc73972e23a, State: Initialized, Role: FOLLOWER
I20260619 01:07:19.278976 8499 consensus_queue.cc:260] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.280850 8499 ts_tablet_manager.cc:1428] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 01:07:19.280864 8501 raft_consensus.cc:348] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.281378 8501 raft_consensus.cc:374] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:19.281499 8501 raft_consensus.cc:729] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f58457c81c74bc79e357da433f98539, State: Initialized, Role: FOLLOWER
I20260619 01:07:19.282228 8501 consensus_queue.cc:260] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.284132 8501 ts_tablet_manager.cc:1428] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260619 01:07:19.284262 8475 heartbeater.cc:503] Master 127.0.97.62:36229 was elected leader, sending a full tablet report...
I20260619 01:07:19.286837 8500 raft_consensus.cc:348] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.287361 8500 raft_consensus.cc:374] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:19.287474 8500 raft_consensus.cc:729] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6fb42b84ca45d5971e7aa1e69a2c01, State: Initialized, Role: FOLLOWER
I20260619 01:07:19.288162 8500 consensus_queue.cc:260] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.289270 8335 heartbeater.cc:503] Master 127.0.97.62:36229 was elected leader, sending a full tablet report...
I20260619 01:07:19.289924 8500 ts_tablet_manager.cc:1428] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Time spent starting tablet: real 0.014s user 0.002s sys 0.011s
W20260619 01:07:19.314849 8336 tablet.cc:2369] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:07:19.373417 8476 tablet.cc:2369] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:07:19.436908 8506 raft_consensus.cc:482] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:07:19.437260 8506 raft_consensus.cc:504] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.439265 8506 leader_election.cc:288] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587), 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745)
I20260619 01:07:19.446331 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.439647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45835 (local address 127.0.97.2:38587)
0619 01:07:19.439891 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:19.439897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:19.439921 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:07:19.440218 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:19.440224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:19.440300 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:19.440413 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:19.440858 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.441841 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.444208 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.445408 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.445556 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.445670 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:19.445890 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 01:07:19.446033 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260619 01:07:19.446352 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.439625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38587 (local address 127.0.97.3:45835)
0619 01:07:19.439814 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:19.439836 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:19.439986 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:19.440431 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:19.440435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:19.440480 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:19.440718 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.440727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.441988 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.441992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.444024 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.444045 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.445564 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.445568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.445728 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.445755 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 01:07:19.445835 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 01:07:19.446018 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1,"spinlock_wait_cycles":23296}
I20260619 01:07:19.446798 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.439899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50667 (local address 127.0.97.1:38745)
0619 01:07:19.440143 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:19.440150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:19.440485 (+ 335us) server_negotiation.cc:407] Connection header received
0619 01:07:19.440703 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:19.440708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:19.440785 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:19.440876 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:19.441463 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.442320 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.444574 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.445784 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.445945 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.446038 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:19.446388 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 01:07:19.446559 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:07:19.446924 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.439626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38745 (local address 127.0.97.3:50667)
0619 01:07:19.440373 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:19.440391 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:19.440566 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:19.440901 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:19.440905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:19.440929 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:19.441259 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.441268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.442470 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.442476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.444424 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.444434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.446030 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.446035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.446214 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.446243 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 01:07:19.446342 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 01:07:19.446485 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":227,"thread_start_us":309,"threads_started":1}
I20260619 01:07:19.448031 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b240c9ade04b45f1ac29319c73c3f84a" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" is_pre_election: true
I20260619 01:07:19.448030 8150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b240c9ade04b45f1ac29319c73c3f84a" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f711ef0abb8448eba326bc73972e23a" is_pre_election: true
I20260619 01:07:19.448640 8150 raft_consensus.cc:2454] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 0.
I20260619 01:07:19.448640 8287 raft_consensus.cc:2454] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 0.
I20260619 01:07:19.449441 8355 leader_election.cc:302] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f58457c81c74bc79e357da433f98539, 9c6fb42b84ca45d5971e7aa1e69a2c01; no voters:
I20260619 01:07:19.449817 8506 raft_consensus.cc:2781] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:07:19.449986 8506 raft_consensus.cc:482] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:07:19.450096 8506 raft_consensus.cc:3037] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:19.452349 8506 raft_consensus.cc:504] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.453027 8506 leader_election.cc:288] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 election: Requested vote from peers 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587), 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745)
I20260619 01:07:19.453593 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b240c9ade04b45f1ac29319c73c3f84a" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01"
I20260619 01:07:19.453671 8150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b240c9ade04b45f1ac29319c73c3f84a" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f711ef0abb8448eba326bc73972e23a"
I20260619 01:07:19.453807 8287 raft_consensus.cc:3037] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:19.453945 8150 raft_consensus.cc:3037] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:19.456251 8287 raft_consensus.cc:2454] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 1.
I20260619 01:07:19.456254 8150 raft_consensus.cc:2454] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 1.
I20260619 01:07:19.456763 8357 leader_election.cc:302] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f711ef0abb8448eba326bc73972e23a, 6f58457c81c74bc79e357da433f98539; no voters:
I20260619 01:07:19.457149 8506 raft_consensus.cc:2781] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:19.457645 8506 raft_consensus.cc:686] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 1 LEADER]: Becoming Leader. State: Replica: 6f58457c81c74bc79e357da433f98539, State: Running, Role: LEADER
I20260619 01:07:19.458238 8506 consensus_queue.cc:237] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.462911 7987 catalog_manager.cc:5495] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f58457c81c74bc79e357da433f98539 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6f58457c81c74bc79e357da433f98539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } health_report { overall_health: HEALTHY } } }
I20260619 01:07:19.484798 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.479562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.0.97.3:42399)
0619 01:07:19.479791 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:19.479796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:19.479813 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:07:19.479993 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:19.479997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:19.480076 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:19.480206 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:07:19.480500 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.481195 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.482609 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.483035 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.483135 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.483597 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:07:19.484204 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:07:19.484315 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:19.484408 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 01:07:19.484480 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 01:07:19.493934 8150 raft_consensus.cc:1261] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Refusing update from remote peer 6f58457c81c74bc79e357da433f98539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:19.493933 8287 raft_consensus.cc:1261] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Refusing update from remote peer 6f58457c81c74bc79e357da433f98539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:19.495060 8509 consensus_queue.cc:1035] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:19.495458 8506 consensus_queue.cc:1035] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:19.508371 8512 mvcc.cc:205] Tried to move back new op lower bound from 7298380756956553216 to 7298380756824662016. Current Snapshot: MvccSnapshot[applied={T|T < 7298380756956553216}]
I20260619 01:07:19.527419 7988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:19.527787 7988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:19.531721 7988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb: Sending DeleteTablet for 3 replicas of tablet b240c9ade04b45f1ac29319c73c3f84a
I20260619 01:07:19.533116 8125 tablet_service.cc:1514] Processing DeleteTablet for tablet b240c9ade04b45f1ac29319c73c3f84a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34888
I20260619 01:07:19.533092 8405 tablet_service.cc:1514] Processing DeleteTablet for tablet b240c9ade04b45f1ac29319c73c3f84a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38078
I20260619 01:07:19.533689 8265 tablet_service.cc:1514] Processing DeleteTablet for tablet b240c9ade04b45f1ac29319c73c3f84a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45026
I20260619 01:07:19.535347 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:07:19.537398 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:07:19.548523 8523 tablet_replica.cc:330] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a: stopping tablet replica
I20260619 01:07:19.549109 8523 raft_consensus.cc:2229] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:19.549736 8523 raft_consensus.cc:2258] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:19.550542 8524 tablet_replica.cc:330] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: stopping tablet replica
I20260619 01:07:19.551039 8524 raft_consensus.cc:2229] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:19.551538 8524 raft_consensus.cc:2258] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:19.551550 8404 tablet_service.cc:1467] Processing CreateTablet for tablet da580679e8ef47c8bdfa49528c1ac759 (DEFAULT_TABLE table=test-table [id=caa218398b524417b93d380a6649f48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:19.552297 8404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da580679e8ef47c8bdfa49528c1ac759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:19.553958 8523 ts_tablet_manager.cc:1905] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:19.558686 8264 tablet_service.cc:1467] Processing CreateTablet for tablet da580679e8ef47c8bdfa49528c1ac759 (DEFAULT_TABLE table=test-table [id=caa218398b524417b93d380a6649f48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:19.559223 8264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da580679e8ef47c8bdfa49528c1ac759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:19.559599 8524 ts_tablet_manager.cc:1905] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:19.560477 8125 tablet_service.cc:1467] Processing CreateTablet for tablet da580679e8ef47c8bdfa49528c1ac759 (DEFAULT_TABLE table=test-table [id=caa218398b524417b93d380a6649f48f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:19.561069 8125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da580679e8ef47c8bdfa49528c1ac759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:19.561281 8522 tablet_replica.cc:330] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: stopping tablet replica
I20260619 01:07:19.561816 8522 raft_consensus.cc:2229] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:07:19.566767 8500 tablet_bootstrap.cc:492] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Bootstrap starting.
I20260619 01:07:19.567138 8522 raft_consensus.cc:2258] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:19.569181 8499 tablet_bootstrap.cc:492] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a: Bootstrap starting.
I20260619 01:07:19.570842 8499 tablet_bootstrap.cc:654] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:19.571420 8501 tablet_bootstrap.cc:492] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539: Bootstrap starting.
I20260619 01:07:19.571521 8522 ts_tablet_manager.cc:1905] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:19.574432 8500 tablet_bootstrap.cc:654] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:19.575383 8501 tablet_bootstrap.cc:654] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:19.576068 8523 ts_tablet_manager.cc:1918] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:19.576330 8523 log.cc:1196] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/b240c9ade04b45f1ac29319c73c3f84a
I20260619 01:07:19.576965 8523 ts_tablet_manager.cc:1939] T b240c9ade04b45f1ac29319c73c3f84a P 5f711ef0abb8448eba326bc73972e23a: Deleting consensus metadata
I20260619 01:07:19.579038 7974 catalog_manager.cc:4841] TS 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745): tablet b240c9ade04b45f1ac29319c73c3f84a (table test-table [id=a7c9df74549f4c8691e004f59a247dfa]) successfully deleted
I20260619 01:07:19.580109 8522 ts_tablet_manager.cc:1918] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:19.580317 8522 log.cc:1196] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/b240c9ade04b45f1ac29319c73c3f84a
I20260619 01:07:19.580915 8522 ts_tablet_manager.cc:1939] T b240c9ade04b45f1ac29319c73c3f84a P 6f58457c81c74bc79e357da433f98539: Deleting consensus metadata
I20260619 01:07:19.581091 8499 tablet_bootstrap.cc:492] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a: No bootstrap required, opened a new log
I20260619 01:07:19.581259 8499 ts_tablet_manager.cc:1397] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260619 01:07:19.582093 8499 raft_consensus.cc:348] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.582341 8499 raft_consensus.cc:374] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:19.582446 8499 raft_consensus.cc:729] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f711ef0abb8448eba326bc73972e23a, State: Initialized, Role: FOLLOWER
I20260619 01:07:19.582753 8499 consensus_queue.cc:260] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.583251 7973 catalog_manager.cc:4841] TS 6f58457c81c74bc79e357da433f98539 (127.0.97.3:42399): tablet b240c9ade04b45f1ac29319c73c3f84a (table test-table [id=a7c9df74549f4c8691e004f59a247dfa]) successfully deleted
I20260619 01:07:19.584105 8499 ts_tablet_manager.cc:1428] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 01:07:19.585886 8500 tablet_bootstrap.cc:492] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01: No bootstrap required, opened a new log
I20260619 01:07:19.586061 8500 ts_tablet_manager.cc:1397] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20260619 01:07:19.586939 8500 raft_consensus.cc:348] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.587177 8500 raft_consensus.cc:374] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:19.587260 8500 raft_consensus.cc:729] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6fb42b84ca45d5971e7aa1e69a2c01, State: Initialized, Role: FOLLOWER
I20260619 01:07:19.587553 8500 consensus_queue.cc:260] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.588766 8500 ts_tablet_manager.cc:1428] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 01:07:19.589267 8501 tablet_bootstrap.cc:492] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539: No bootstrap required, opened a new log
I20260619 01:07:19.592496 8501 ts_tablet_manager.cc:1397] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20260619 01:07:19.593514 8501 raft_consensus.cc:348] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.593796 8501 raft_consensus.cc:374] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:19.593904 8501 raft_consensus.cc:729] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f58457c81c74bc79e357da433f98539, State: Initialized, Role: FOLLOWER
I20260619 01:07:19.594214 8501 consensus_queue.cc:260] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.596261 8524 ts_tablet_manager.cc:1918] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:19.597677 8501 ts_tablet_manager.cc:1428] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260619 01:07:19.598088 8524 log.cc:1196] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/b240c9ade04b45f1ac29319c73c3f84a
I20260619 01:07:19.598815 8524 ts_tablet_manager.cc:1939] T b240c9ade04b45f1ac29319c73c3f84a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting consensus metadata
I20260619 01:07:19.600971 7972 catalog_manager.cc:4841] TS 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587): tablet b240c9ade04b45f1ac29319c73c3f84a (table test-table [id=a7c9df74549f4c8691e004f59a247dfa]) successfully deleted
I20260619 01:07:19.683677 8509 raft_consensus.cc:482] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:07:19.683945 8509 raft_consensus.cc:504] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.684619 8509 leader_election.cc:288] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745), 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587)
I20260619 01:07:19.685246 8150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da580679e8ef47c8bdfa49528c1ac759" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f711ef0abb8448eba326bc73972e23a" is_pre_election: true
I20260619 01:07:19.685258 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da580679e8ef47c8bdfa49528c1ac759" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" is_pre_election: true
I20260619 01:07:19.685549 8150 raft_consensus.cc:2454] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 0.
I20260619 01:07:19.685567 8287 raft_consensus.cc:2454] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 0.
I20260619 01:07:19.686031 8357 leader_election.cc:302] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f711ef0abb8448eba326bc73972e23a, 6f58457c81c74bc79e357da433f98539; no voters:
I20260619 01:07:19.686429 8509 raft_consensus.cc:2781] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:07:19.686568 8509 raft_consensus.cc:482] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:07:19.686712 8509 raft_consensus.cc:3037] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:19.689102 8509 raft_consensus.cc:504] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.689814 8509 leader_election.cc:288] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 election: Requested vote from peers 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745), 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587)
I20260619 01:07:19.690423 8150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da580679e8ef47c8bdfa49528c1ac759" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f711ef0abb8448eba326bc73972e23a"
I20260619 01:07:19.690423 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da580679e8ef47c8bdfa49528c1ac759" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01"
I20260619 01:07:19.690652 8150 raft_consensus.cc:3037] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:19.690652 8287 raft_consensus.cc:3037] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:19.693814 8287 raft_consensus.cc:2454] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 1.
I20260619 01:07:19.693814 8150 raft_consensus.cc:2454] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 1.
I20260619 01:07:19.694350 8355 leader_election.cc:302] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5f711ef0abb8448eba326bc73972e23a, 6f58457c81c74bc79e357da433f98539, 9c6fb42b84ca45d5971e7aa1e69a2c01; no voters:
I20260619 01:07:19.694734 8509 raft_consensus.cc:2781] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:19.694977 8509 raft_consensus.cc:686] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 1 LEADER]: Becoming Leader. State: Replica: 6f58457c81c74bc79e357da433f98539, State: Running, Role: LEADER
I20260619 01:07:19.695294 8509 consensus_queue.cc:237] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.698318 7987 catalog_manager.cc:5495] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f58457c81c74bc79e357da433f98539 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6f58457c81c74bc79e357da433f98539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } health_report { overall_health: HEALTHY } } }
W20260619 01:07:19.764227 8196 tablet.cc:2369] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 01:07:19 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831237, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:19.774654 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.755589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36229 (local address 127.0.0.1:37120)
0619 01:07:19.756511 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:19.756535 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:19.756678 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:19.757074 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:19.757077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:19.757832 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:19.758034 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.758045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.759496 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.759501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.760433 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:19.760444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.761454 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.761459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:19.761590 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.765720 (+ 4130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:19.765743 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:19.768087 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:19.772134 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:19.772149 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:19.772155 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:19.772606 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:19.773027 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:19.773030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:19.773032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:19.773171 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:19.773852 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:19.773858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:19.774022 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 01:07:19.774161 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 01:07:19.774387 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":621,"threads_started":1}
I20260619 01:07:19.774983 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:19.755714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.0.97.62:36229)
0619 01:07:19.756279 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:19.756285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:19.756634 (+ 349us) server_negotiation.cc:407] Connection header received
0619 01:07:19.756805 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:19.756811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:19.756889 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:19.757016 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:19.758220 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.759334 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.760597 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:19.761237 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:19.761400 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:19.768362 (+ 6962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:19.768369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:19.768374 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:19.768419 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:19.771954 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:19.772744 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:19.772749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:19.772753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:19.772902 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:19.773277 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:19.773284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:19.773315 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:19.773711 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:19.773885 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:19.774295 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 01:07:19.774524 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":287,"threads_started":1}
I20260619 01:07:19.780712 7987 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37120:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 01:07:19.781854 7987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:07:19.789618 8125 tablet_service.cc:1467] Processing CreateTablet for tablet e6817ac2587740c38b06b70bcc687182 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d321d6506d1d45b0a74636a01dd0c723]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:07:19.789822 8404 tablet_service.cc:1467] Processing CreateTablet for tablet e6817ac2587740c38b06b70bcc687182 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d321d6506d1d45b0a74636a01dd0c723]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:07:19.790040 8264 tablet_service.cc:1467] Processing CreateTablet for tablet e6817ac2587740c38b06b70bcc687182 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d321d6506d1d45b0a74636a01dd0c723]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:07:19.790256 8125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6817ac2587740c38b06b70bcc687182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:19.790396 8404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6817ac2587740c38b06b70bcc687182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:19.790505 8264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6817ac2587740c38b06b70bcc687182. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:19.798408 8500 tablet_bootstrap.cc:492] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Bootstrap starting.
I20260619 01:07:19.800010 8500 tablet_bootstrap.cc:654] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:19.800227 8499 tablet_bootstrap.cc:492] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a: Bootstrap starting.
I20260619 01:07:19.801347 8501 tablet_bootstrap.cc:492] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: Bootstrap starting.
I20260619 01:07:19.802209 8499 tablet_bootstrap.cc:654] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:19.803355 8501 tablet_bootstrap.cc:654] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:19.804714 8499 tablet_bootstrap.cc:492] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a: No bootstrap required, opened a new log
I20260619 01:07:19.804963 8499 ts_tablet_manager.cc:1397] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 01:07:19.805984 8499 raft_consensus.cc:348] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.806428 8499 raft_consensus.cc:374] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:19.806578 8499 raft_consensus.cc:729] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f711ef0abb8448eba326bc73972e23a, State: Initialized, Role: FOLLOWER
I20260619 01:07:19.806955 8501 tablet_bootstrap.cc:492] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: No bootstrap required, opened a new log
I20260619 01:07:19.807158 8501 ts_tablet_manager.cc:1397] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 01:07:19.807119 8499 consensus_queue.cc:260] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.807955 8501 raft_consensus.cc:348] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.807906 8505 tablet_replica.cc:439] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } } }
I20260619 01:07:19.808274 8501 raft_consensus.cc:374] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:19.808295 8505 tablet_replica.cc:442] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:07:19.808364 8501 raft_consensus.cc:729] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f58457c81c74bc79e357da433f98539, State: Initialized, Role: FOLLOWER
I20260619 01:07:19.808667 8501 consensus_queue.cc:260] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.810051 8501 ts_tablet_manager.cc:1428] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:19.809940 8509 tablet_replica.cc:439] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } } }
I20260619 01:07:19.810262 8509 tablet_replica.cc:442] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:07:19.813774 8499 ts_tablet_manager.cc:1428] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20260619 01:07:19.814895 8500 tablet_bootstrap.cc:492] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01: No bootstrap required, opened a new log
I20260619 01:07:19.816361 8500 ts_tablet_manager.cc:1397] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.002s
I20260619 01:07:19.817065 8500 raft_consensus.cc:348] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.817297 8500 raft_consensus.cc:374] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:19.817426 8500 raft_consensus.cc:729] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6fb42b84ca45d5971e7aa1e69a2c01, State: Initialized, Role: FOLLOWER
I20260619 01:07:19.817713 8500 consensus_queue.cc:260] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:19.818303 8500 ts_tablet_manager.cc:1428] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 01:07:19.818161 8507 tablet_replica.cc:439] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } } }
I20260619 01:07:19.818619 8507 tablet_replica.cc:442] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:07:20.150341 8509 consensus_queue.cc:1035] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:20.153123 8509 raft_consensus.cc:482] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:07:20.153393 8509 raft_consensus.cc:504] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.154095 8509 leader_election.cc:288] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745), 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587)
I20260619 01:07:20.154891 8150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6817ac2587740c38b06b70bcc687182" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f711ef0abb8448eba326bc73972e23a" is_pre_election: true
I20260619 01:07:20.155063 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6817ac2587740c38b06b70bcc687182" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" is_pre_election: true
I20260619 01:07:20.155169 8150 raft_consensus.cc:2454] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 0.
I20260619 01:07:20.155325 8287 raft_consensus.cc:2454] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 0.
I20260619 01:07:20.155761 8355 leader_election.cc:302] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f58457c81c74bc79e357da433f98539, 9c6fb42b84ca45d5971e7aa1e69a2c01; no voters:
I20260619 01:07:20.156046 8540 raft_consensus.cc:2781] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:07:20.156145 8540 raft_consensus.cc:482] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:07:20.156338 8540 raft_consensus.cc:3037] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:20.156744 8506 consensus_queue.cc:1035] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:20.159385 8540 raft_consensus.cc:504] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.160076 8540 leader_election.cc:288] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 election: Requested vote from peers 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745), 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587)
I20260619 01:07:20.160475 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6817ac2587740c38b06b70bcc687182" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01"
I20260619 01:07:20.160465 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6817ac2587740c38b06b70bcc687182" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f711ef0abb8448eba326bc73972e23a"
I20260619 01:07:20.160688 8149 raft_consensus.cc:3037] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:20.160713 8287 raft_consensus.cc:3037] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:20.162930 8287 raft_consensus.cc:2454] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 1.
I20260619 01:07:20.163331 8355 leader_election.cc:302] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f58457c81c74bc79e357da433f98539, 9c6fb42b84ca45d5971e7aa1e69a2c01; no voters:
I20260619 01:07:20.163616 8540 raft_consensus.cc:2781] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:20.163626 8149 raft_consensus.cc:2454] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 1.
I20260619 01:07:20.163848 8540 raft_consensus.cc:686] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [term 1 LEADER]: Becoming Leader. State: Replica: 6f58457c81c74bc79e357da433f98539, State: Running, Role: LEADER
I20260619 01:07:20.164381 8540 consensus_queue.cc:237] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.165930 8509 tablet_replica.cc:439] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: TxnStatusTablet state changed. Reason: New leader 6f58457c81c74bc79e357da433f98539. Latest consensus state: current_term: 1 leader_uuid: "6f58457c81c74bc79e357da433f98539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } } }
I20260619 01:07:20.166226 8509 tablet_replica.cc:442] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:07:20.166779 8546 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 01:07:20.167011 8546 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 01:07:20.167176 7988 catalog_manager.cc:5495] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f58457c81c74bc79e357da433f98539 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6f58457c81c74bc79e357da433f98539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } health_report { overall_health: HEALTHY } } }
I20260619 01:07:20.266587 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.260049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42399 (local address 127.0.0.1:38088)
0619 01:07:20.260352 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:20.260375 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:20.260527 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:20.261213 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:20.261218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:20.261232 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:07:20.261472 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:20.261488 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.262672 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.262678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:20.264342 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:20.264361 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.265236 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.265242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:20.265462 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.265527 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:07:20.265977 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:07:20.266003 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 01:07:20.266096 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 01:07:20.266261 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 01:07:20.266646 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.260214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.0.97.3:42399)
0619 01:07:20.260911 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:20.260915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:20.260936 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:07:20.260995 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:20.260998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:20.261058 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:20.261144 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:07:20.261649 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.262527 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.264509 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.264997 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.265217 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.265746 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:07:20.265873 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:07:20.265977 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:20.266121 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 01:07:20.266209 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":404,"threads_started":1}
I20260619 01:07:20.270645 8149 raft_consensus.cc:1261] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Refusing update from remote peer 6f58457c81c74bc79e357da433f98539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:20.270666 8287 raft_consensus.cc:1261] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Refusing update from remote peer 6f58457c81c74bc79e357da433f98539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:20.271271 8509 consensus_queue.cc:1035] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:20.271549 8540 consensus_queue.cc:1035] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:20.275453 8507 tablet_replica.cc:439] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01: TxnStatusTablet state changed. Reason: New leader 6f58457c81c74bc79e357da433f98539. Latest consensus state: current_term: 1 leader_uuid: "6f58457c81c74bc79e357da433f98539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } } }
I20260619 01:07:20.275794 8507 tablet_replica.cc:442] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:07:20.276360 8505 tablet_replica.cc:439] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a: TxnStatusTablet state changed. Reason: New leader 6f58457c81c74bc79e357da433f98539. Latest consensus state: current_term: 1 leader_uuid: "6f58457c81c74bc79e357da433f98539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } } }
I20260619 01:07:20.276646 8505 tablet_replica.cc:442] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:07:20.280205 8544 mvcc.cc:205] Tried to move back new op lower bound from 7298380760142204928 to 7298380759715692544. Current Snapshot: MvccSnapshot[applied={T|T < 7298380760142204928}]
I20260619 01:07:20.281390 8542 mvcc.cc:205] Tried to move back new op lower bound from 7298380760142204928 to 7298380759715692544. Current Snapshot: MvccSnapshot[applied={T|T < 7298380760142204928}]
I20260619 01:07:20.281788 8509 tablet_replica.cc:439] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f58457c81c74bc79e357da433f98539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } } }
I20260619 01:07:20.282084 8509 tablet_replica.cc:442] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:07:20.282308 8509 tablet_replica.cc:439] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f58457c81c74bc79e357da433f98539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } } }
I20260619 01:07:20.282496 8509 tablet_replica.cc:442] T e6817ac2587740c38b06b70bcc687182 P 6f58457c81c74bc79e357da433f98539: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:07:20.282948 8505 tablet_replica.cc:439] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f58457c81c74bc79e357da433f98539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } } }
I20260619 01:07:20.283180 8505 tablet_replica.cc:442] T e6817ac2587740c38b06b70bcc687182 P 5f711ef0abb8448eba326bc73972e23a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:07:20.287060 8507 tablet_replica.cc:439] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f58457c81c74bc79e357da433f98539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } } }
I20260619 01:07:20.287338 8507 tablet_replica.cc:442] T e6817ac2587740c38b06b70bcc687182 P 9c6fb42b84ca45d5971e7aa1e69a2c01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:07:20.307106 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.300107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49931 (local address 127.0.97.3:42399)
0619 01:07:20.300331 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:20.300339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:20.301007 (+ 668us) server_negotiation.cc:407] Connection header received
0619 01:07:20.301326 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:20.301333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:20.301406 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:20.301492 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:07:20.302187 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.303032 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.305226 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.305694 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.305788 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.306364 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:07:20.306465 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:07:20.306527 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:20.306694 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 01:07:20.306823 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":62}
I20260619 01:07:20.307490 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.300091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42399 (local address 127.0.97.3:49931)
0619 01:07:20.300856 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:20.300879 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:20.301095 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:20.301524 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:20.301530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:20.301544 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:07:20.301972 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:20.301991 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.303154 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.303162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:20.305082 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:20.305098 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.305869 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.305877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:20.306086 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.306173 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:07:20.306560 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:07:20.306572 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 01:07:20.306670 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 01:07:20.306904 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":4,"thread_start_us":408,"threads_started":1}
I20260619 01:07:20.385149 7988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:20.385453 7988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:20.388638 7988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb: Sending DeleteTablet for 3 replicas of tablet da580679e8ef47c8bdfa49528c1ac759
I20260619 01:07:20.389539 8125 tablet_service.cc:1514] Processing DeleteTablet for tablet da580679e8ef47c8bdfa49528c1ac759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34888
I20260619 01:07:20.390058 8264 tablet_service.cc:1514] Processing DeleteTablet for tablet da580679e8ef47c8bdfa49528c1ac759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45026
I20260619 01:07:20.393903 8405 tablet_service.cc:1514] Processing DeleteTablet for tablet da580679e8ef47c8bdfa49528c1ac759 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38078
I20260619 01:07:20.401825 8561 tablet_replica.cc:330] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539: stopping tablet replica
I20260619 01:07:20.402292 8561 raft_consensus.cc:2229] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:07:20.403007 8561 raft_consensus.cc:2258] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:20.404551 8561 ts_tablet_manager.cc:1905] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:20.406023 8560 tablet_replica.cc:330] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01: stopping tablet replica
I20260619 01:07:20.406255 8560 raft_consensus.cc:2229] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:20.407379 8559 tablet_replica.cc:330] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a: stopping tablet replica
I20260619 01:07:20.407446 8560 raft_consensus.cc:2258] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:20.407648 8559 raft_consensus.cc:2229] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:20.407917 8559 raft_consensus.cc:2258] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:20.408854 8560 ts_tablet_manager.cc:1905] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:20.408985 8559 ts_tablet_manager.cc:1905] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:20.413062 8561 ts_tablet_manager.cc:1918] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:07:20.413417 8561 log.cc:1196] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/da580679e8ef47c8bdfa49528c1ac759
I20260619 01:07:20.414029 8561 ts_tablet_manager.cc:1939] T da580679e8ef47c8bdfa49528c1ac759 P 6f58457c81c74bc79e357da433f98539: Deleting consensus metadata
I20260619 01:07:20.415298 7973 catalog_manager.cc:4841] TS 6f58457c81c74bc79e357da433f98539 (127.0.97.3:42399): tablet da580679e8ef47c8bdfa49528c1ac759 (table test-table [id=caa218398b524417b93d380a6649f48f]) successfully deleted
I20260619 01:07:20.416235 8559 ts_tablet_manager.cc:1918] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:07:20.416384 8559 log.cc:1196] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/da580679e8ef47c8bdfa49528c1ac759
I20260619 01:07:20.416774 8559 ts_tablet_manager.cc:1939] T da580679e8ef47c8bdfa49528c1ac759 P 5f711ef0abb8448eba326bc73972e23a: Deleting consensus metadata
I20260619 01:07:20.417905 8560 ts_tablet_manager.cc:1918] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:07:20.418061 8560 log.cc:1196] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/da580679e8ef47c8bdfa49528c1ac759
I20260619 01:07:20.418499 8560 ts_tablet_manager.cc:1939] T da580679e8ef47c8bdfa49528c1ac759 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting consensus metadata
I20260619 01:07:20.419728 7972 catalog_manager.cc:4841] TS 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587): tablet da580679e8ef47c8bdfa49528c1ac759 (table test-table [id=caa218398b524417b93d380a6649f48f]) successfully deleted
I20260619 01:07:20.425688 7974 catalog_manager.cc:4841] TS 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745): tablet da580679e8ef47c8bdfa49528c1ac759 (table test-table [id=caa218398b524417b93d380a6649f48f]) successfully deleted
Jun 19 01:07:20 dist-test-slave-5p8w krb5kdc[7943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 01:07:20 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:20 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 01:07:20.466946 8569 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:20.456133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.0.97.62:36229)
0619 01:07:20.456974 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:20.456979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:20.456995 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:07:20.457044 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:20.457048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:20.457112 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:20.457228 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:20.458564 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.459366 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.460642 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.461052 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.461182 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.466569 (+ 5387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37126: BlockingRecv error: recv got EOF from 127.0.0.1:37126 (error 108)
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":512,"threads_started":1}
Jun 19 01:07:20 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:07:20.486019 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.469688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.0.97.62:36229)
0619 01:07:20.469872 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:20.469880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:20.470109 (+ 229us) server_negotiation.cc:407] Connection header received
0619 01:07:20.470273 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:20.470280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:20.470372 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:20.470501 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:20.471779 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.472969 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.474269 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.474992 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.475152 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.479189 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:20.479198 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:20.479204 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:20.479258 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:20.483047 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:20.483765 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:20.483773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:20.483776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:20.483980 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:20.484392 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:20.484401 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:20.484405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:20.484832 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:20.485143 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:20.485365 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 01:07:20.485619 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 01:07:20.489954 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:07:20.490784 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:07:20.498507 8125 tablet_service.cc:1467] Processing CreateTablet for tablet d1e74c0df9f7454fb25e38d119bf808a (DEFAULT_TABLE table=test-table [id=cc9721b3781342bd914a7e4bed00085f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:20.498654 8264 tablet_service.cc:1467] Processing CreateTablet for tablet d1e74c0df9f7454fb25e38d119bf808a (DEFAULT_TABLE table=test-table [id=cc9721b3781342bd914a7e4bed00085f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:20.498967 8125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e74c0df9f7454fb25e38d119bf808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:20.499164 8264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e74c0df9f7454fb25e38d119bf808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:20.502158 8405 tablet_service.cc:1467] Processing CreateTablet for tablet d1e74c0df9f7454fb25e38d119bf808a (DEFAULT_TABLE table=test-table [id=cc9721b3781342bd914a7e4bed00085f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:20.502605 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e74c0df9f7454fb25e38d119bf808a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:20.506726 8575 tablet_bootstrap.cc:492] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Bootstrap starting.
I20260619 01:07:20.508606 8575 tablet_bootstrap.cc:654] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:20.511085 8575 tablet_bootstrap.cc:492] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01: No bootstrap required, opened a new log
I20260619 01:07:20.511264 8575 ts_tablet_manager.cc:1397] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 01:07:20.512063 8575 raft_consensus.cc:348] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.512303 8575 raft_consensus.cc:374] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:20.512396 8575 raft_consensus.cc:729] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6fb42b84ca45d5971e7aa1e69a2c01, State: Initialized, Role: FOLLOWER
I20260619 01:07:20.513731 8575 consensus_queue.cc:260] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.515056 8575 ts_tablet_manager.cc:1428] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 01:07:20.521529 8578 tablet_bootstrap.cc:492] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a: Bootstrap starting.
I20260619 01:07:20.522131 8577 tablet_bootstrap.cc:492] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539: Bootstrap starting.
I20260619 01:07:20.523448 8578 tablet_bootstrap.cc:654] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:20.524209 8577 tablet_bootstrap.cc:654] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:20.526554 8578 tablet_bootstrap.cc:492] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a: No bootstrap required, opened a new log
I20260619 01:07:20.526713 8578 ts_tablet_manager.cc:1397] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 01:07:20.527002 8577 tablet_bootstrap.cc:492] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539: No bootstrap required, opened a new log
I20260619 01:07:20.527179 8577 ts_tablet_manager.cc:1397] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 01:07:20.527541 8578 raft_consensus.cc:348] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.527808 8578 raft_consensus.cc:374] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:20.527905 8578 raft_consensus.cc:729] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f711ef0abb8448eba326bc73972e23a, State: Initialized, Role: FOLLOWER
I20260619 01:07:20.528142 8578 consensus_queue.cc:260] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.528151 8577 raft_consensus.cc:348] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.528445 8577 raft_consensus.cc:374] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:20.528538 8577 raft_consensus.cc:729] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f58457c81c74bc79e357da433f98539, State: Initialized, Role: FOLLOWER
I20260619 01:07:20.528946 8578 ts_tablet_manager.cc:1428] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 01:07:20.528834 8577 consensus_queue.cc:260] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.530027 8577 ts_tablet_manager.cc:1428] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:20.540463 8505 raft_consensus.cc:482] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:07:20.540745 8505 raft_consensus.cc:504] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.542390 8505 leader_election.cc:288] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587), 6f58457c81c74bc79e357da433f98539 (127.0.97.3:42399)
I20260619 01:07:20.557744 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.543396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37861 (local address 127.0.97.3:42399)
0619 01:07:20.543669 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:20.543679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:20.550708 (+ 7029us) server_negotiation.cc:407] Connection header received
0619 01:07:20.550932 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:20.550937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:20.550994 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:20.551092 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:20.551738 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.552972 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.555255 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.556110 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.556275 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.556389 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:20.557068 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 01:07:20.557403 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 01:07:20.558584 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.542894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42399 (local address 127.0.97.1:37861)
0619 01:07:20.550575 (+ 7681us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:20.550595 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:20.550732 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:20.551104 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:20.551109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:20.551131 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:20.551550 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:20.551563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.553153 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.553158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:20.555110 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:20.555120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.556320 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.556326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:20.556741 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.556822 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 01:07:20.558023 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0619 01:07:20.558185 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7558,"thread_start_us":6619,"threads_started":1}
I20260619 01:07:20.558637 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.542820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38587 (local address 127.0.97.1:43643)
0619 01:07:20.549962 (+ 7142us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:20.549996 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:20.550209 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:20.550823 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:20.550829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:20.550860 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:20.551349 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:20.551365 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.552689 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.552694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:20.554853 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:20.554868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.557522 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.557529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:20.557703 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.557747 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 01:07:20.558152 (+ 405us) client_negotiation.cc:240] Negotiation successful
0619 01:07:20.558264 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6912,"mutex_wait_us":1,"thread_start_us":7430,"threads_started":1}
I20260619 01:07:20.559058 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.542895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43643 (local address 127.0.97.2:38587)
0619 01:07:20.550327 (+ 7432us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:20.550334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:20.550363 (+ 29us) server_negotiation.cc:407] Connection header received
0619 01:07:20.550442 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:20.550447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:20.550524 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:20.550637 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:20.551602 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.552541 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.555042 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.556362 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.556517 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.558414 (+ 1897us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:20.558482 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 01:07:20.558646 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7276,"thread_start_us":7106,"threads_started":1}
I20260619 01:07:20.559501 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e74c0df9f7454fb25e38d119bf808a" candidate_uuid: "5f711ef0abb8448eba326bc73972e23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f58457c81c74bc79e357da433f98539" is_pre_election: true
I20260619 01:07:20.559721 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e74c0df9f7454fb25e38d119bf808a" candidate_uuid: "5f711ef0abb8448eba326bc73972e23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" is_pre_election: true
I20260619 01:07:20.559981 8287 raft_consensus.cc:2454] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f711ef0abb8448eba326bc73972e23a in term 0.
I20260619 01:07:20.560052 8425 raft_consensus.cc:2454] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f711ef0abb8448eba326bc73972e23a in term 0.
I20260619 01:07:20.560685 8075 leader_election.cc:302] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f711ef0abb8448eba326bc73972e23a, 9c6fb42b84ca45d5971e7aa1e69a2c01; no voters:
I20260619 01:07:20.561070 8505 raft_consensus.cc:2781] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:07:20.561158 8505 raft_consensus.cc:482] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:07:20.561218 8505 raft_consensus.cc:3037] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:20.563431 8505 raft_consensus.cc:504] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.564162 8505 leader_election.cc:288] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [CANDIDATE]: Term 1 election: Requested vote from peers 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587), 6f58457c81c74bc79e357da433f98539 (127.0.97.3:42399)
I20260619 01:07:20.564682 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e74c0df9f7454fb25e38d119bf808a" candidate_uuid: "5f711ef0abb8448eba326bc73972e23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01"
I20260619 01:07:20.564944 8287 raft_consensus.cc:3037] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:20.567518 8287 raft_consensus.cc:2454] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f711ef0abb8448eba326bc73972e23a in term 1.
I20260619 01:07:20.567991 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e74c0df9f7454fb25e38d119bf808a" candidate_uuid: "5f711ef0abb8448eba326bc73972e23a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f58457c81c74bc79e357da433f98539"
I20260619 01:07:20.568233 8425 raft_consensus.cc:3037] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:20.571017 8425 raft_consensus.cc:2454] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f711ef0abb8448eba326bc73972e23a in term 1.
I20260619 01:07:20.571569 8076 leader_election.cc:302] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f711ef0abb8448eba326bc73972e23a, 6f58457c81c74bc79e357da433f98539; no voters:
I20260619 01:07:20.571920 8505 raft_consensus.cc:2781] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:20.572412 8505 raft_consensus.cc:686] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 1 LEADER]: Becoming Leader. State: Replica: 5f711ef0abb8448eba326bc73972e23a, State: Running, Role: LEADER
I20260619 01:07:20.572914 8505 consensus_queue.cc:237] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.576785 7988 catalog_manager.cc:5495] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f711ef0abb8448eba326bc73972e23a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5f711ef0abb8448eba326bc73972e23a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } health_report { overall_health: UNKNOWN } } }
I20260619 01:07:20.599566 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.591984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.0.97.1:38745)
0619 01:07:20.593759 (+ 1775us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:20.593763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:20.593782 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:07:20.593865 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:20.593868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:20.593939 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:20.594106 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:07:20.594541 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.595632 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.597188 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.597712 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.597840 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.598285 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:07:20.598893 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:07:20.599066 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:20.599118 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 01:07:20.599213 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663,"thread_start_us":1534,"threads_started":1}
I20260619 01:07:20.604275 8287 raft_consensus.cc:1261] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Refusing update from remote peer 5f711ef0abb8448eba326bc73972e23a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:20.605341 8425 raft_consensus.cc:1261] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Refusing update from remote peer 5f711ef0abb8448eba326bc73972e23a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:20.606163 8584 consensus_queue.cc:1035] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:20.606590 8505 consensus_queue.cc:1035] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:20.626901 7987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:20.627147 7987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:20.630398 7987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb: Sending DeleteTablet for 3 replicas of tablet d1e74c0df9f7454fb25e38d119bf808a
I20260619 01:07:20.631306 8125 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e74c0df9f7454fb25e38d119bf808a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34888
I20260619 01:07:20.631546 8264 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e74c0df9f7454fb25e38d119bf808a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45026
I20260619 01:07:20.631630 8559 tablet_replica.cc:330] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a: stopping tablet replica
I20260619 01:07:20.631856 8559 raft_consensus.cc:2229] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:07:20.632436 8559 raft_consensus.cc:2258] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:20.632426 8405 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e74c0df9f7454fb25e38d119bf808a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38078
I20260619 01:07:20.632701 8561 tablet_replica.cc:330] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539: stopping tablet replica
I20260619 01:07:20.632741 8560 tablet_replica.cc:330] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01: stopping tablet replica
I20260619 01:07:20.632902 8561 raft_consensus.cc:2229] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:20.632926 8560 raft_consensus.cc:2229] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:20.633158 8560 raft_consensus.cc:2258] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:20.633188 8561 raft_consensus.cc:2258] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:20.633502 7987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:07:20.634245 7987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:07:20.634614 8559 ts_tablet_manager.cc:1905] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:20.634815 8561 ts_tablet_manager.cc:1905] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:20.642445 8559 ts_tablet_manager.cc:1918] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:20.642629 8559 log.cc:1196] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/d1e74c0df9f7454fb25e38d119bf808a
I20260619 01:07:20.643067 8559 ts_tablet_manager.cc:1939] T d1e74c0df9f7454fb25e38d119bf808a P 5f711ef0abb8448eba326bc73972e23a: Deleting consensus metadata
I20260619 01:07:20.644213 8264 tablet_service.cc:1467] Processing CreateTablet for tablet bc9af01195b5477ea67b340ddde2d781 (DEFAULT_TABLE table=test-table [id=23c08d083a4346e9b81cfaefd1e30e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:20.644659 8264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9af01195b5477ea67b340ddde2d781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:20.646203 8560 ts_tablet_manager.cc:1905] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:20.650190 8125 tablet_service.cc:1467] Processing CreateTablet for tablet bc9af01195b5477ea67b340ddde2d781 (DEFAULT_TABLE table=test-table [id=23c08d083a4346e9b81cfaefd1e30e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:20.650718 8125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9af01195b5477ea67b340ddde2d781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:20.650830 8405 tablet_service.cc:1467] Processing CreateTablet for tablet bc9af01195b5477ea67b340ddde2d781 (DEFAULT_TABLE table=test-table [id=23c08d083a4346e9b81cfaefd1e30e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:20.651259 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9af01195b5477ea67b340ddde2d781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:20.653271 8560 ts_tablet_manager.cc:1918] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:20.653460 8560 log.cc:1196] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/d1e74c0df9f7454fb25e38d119bf808a
I20260619 01:07:20.653852 8560 ts_tablet_manager.cc:1939] T d1e74c0df9f7454fb25e38d119bf808a P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting consensus metadata
I20260619 01:07:20.655102 7972 catalog_manager.cc:4841] TS 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587): tablet d1e74c0df9f7454fb25e38d119bf808a (table test-table [id=cc9721b3781342bd914a7e4bed00085f]) successfully deleted
I20260619 01:07:20.656787 7974 catalog_manager.cc:4841] TS 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745): tablet d1e74c0df9f7454fb25e38d119bf808a (table test-table [id=cc9721b3781342bd914a7e4bed00085f]) successfully deleted
I20260619 01:07:20.658661 8578 tablet_bootstrap.cc:492] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a: Bootstrap starting.
I20260619 01:07:20.660557 8578 tablet_bootstrap.cc:654] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:20.661362 8561 ts_tablet_manager.cc:1918] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:20.661526 8561 log.cc:1196] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/d1e74c0df9f7454fb25e38d119bf808a
I20260619 01:07:20.661803 8575 tablet_bootstrap.cc:492] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Bootstrap starting.
I20260619 01:07:20.661989 8561 ts_tablet_manager.cc:1939] T d1e74c0df9f7454fb25e38d119bf808a P 6f58457c81c74bc79e357da433f98539: Deleting consensus metadata
I20260619 01:07:20.663266 7973 catalog_manager.cc:4841] TS 6f58457c81c74bc79e357da433f98539 (127.0.97.3:42399): tablet d1e74c0df9f7454fb25e38d119bf808a (table test-table [id=cc9721b3781342bd914a7e4bed00085f]) successfully deleted
I20260619 01:07:20.663545 8575 tablet_bootstrap.cc:654] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:20.665503 8577 tablet_bootstrap.cc:492] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539: Bootstrap starting.
I20260619 01:07:20.667179 8577 tablet_bootstrap.cc:654] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:20.670857 8578 tablet_bootstrap.cc:492] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a: No bootstrap required, opened a new log
I20260619 01:07:20.671092 8578 ts_tablet_manager.cc:1397] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20260619 01:07:20.671816 8578 raft_consensus.cc:348] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.672036 8578 raft_consensus.cc:374] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:20.672122 8578 raft_consensus.cc:729] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f711ef0abb8448eba326bc73972e23a, State: Initialized, Role: FOLLOWER
I20260619 01:07:20.672405 8578 consensus_queue.cc:260] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.673446 8577 tablet_bootstrap.cc:492] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539: No bootstrap required, opened a new log
I20260619 01:07:20.673645 8577 ts_tablet_manager.cc:1397] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260619 01:07:20.674279 8577 raft_consensus.cc:348] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.674496 8577 raft_consensus.cc:374] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:20.674589 8577 raft_consensus.cc:729] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f58457c81c74bc79e357da433f98539, State: Initialized, Role: FOLLOWER
I20260619 01:07:20.674858 8577 consensus_queue.cc:260] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.675420 8577 ts_tablet_manager.cc:1428] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 01:07:20.676110 8578 ts_tablet_manager.cc:1428] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260619 01:07:20.678360 8575 tablet_bootstrap.cc:492] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01: No bootstrap required, opened a new log
I20260619 01:07:20.678632 8575 ts_tablet_manager.cc:1397] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260619 01:07:20.679486 8575 raft_consensus.cc:348] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.679912 8575 raft_consensus.cc:374] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:20.680044 8575 raft_consensus.cc:729] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6fb42b84ca45d5971e7aa1e69a2c01, State: Initialized, Role: FOLLOWER
I20260619 01:07:20.680380 8575 consensus_queue.cc:260] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.681090 8575 ts_tablet_manager.cc:1428] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:07:20.737053 8540 raft_consensus.cc:482] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:07:20.737358 8540 raft_consensus.cc:504] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.738165 8540 leader_election.cc:288] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745), 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587)
I20260619 01:07:20.738801 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9af01195b5477ea67b340ddde2d781" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" is_pre_election: true
I20260619 01:07:20.738822 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9af01195b5477ea67b340ddde2d781" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f711ef0abb8448eba326bc73972e23a" is_pre_election: true
I20260619 01:07:20.739102 8287 raft_consensus.cc:2454] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 0.
I20260619 01:07:20.739245 8149 raft_consensus.cc:2454] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 0.
I20260619 01:07:20.739629 8355 leader_election.cc:302] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f58457c81c74bc79e357da433f98539, 9c6fb42b84ca45d5971e7aa1e69a2c01; no voters:
I20260619 01:07:20.739967 8540 raft_consensus.cc:2781] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:07:20.740080 8540 raft_consensus.cc:482] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:07:20.740213 8540 raft_consensus.cc:3037] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:20.742771 8540 raft_consensus.cc:504] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.743520 8540 leader_election.cc:288] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 election: Requested vote from peers 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745), 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587)
I20260619 01:07:20.744032 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9af01195b5477ea67b340ddde2d781" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f711ef0abb8448eba326bc73972e23a"
I20260619 01:07:20.744282 8149 raft_consensus.cc:3037] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:20.744339 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9af01195b5477ea67b340ddde2d781" candidate_uuid: "6f58457c81c74bc79e357da433f98539" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01"
I20260619 01:07:20.744591 8287 raft_consensus.cc:3037] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:20.746898 8149 raft_consensus.cc:2454] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 1.
I20260619 01:07:20.747416 8357 leader_election.cc:302] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f711ef0abb8448eba326bc73972e23a, 6f58457c81c74bc79e357da433f98539; no voters:
I20260619 01:07:20.747773 8540 raft_consensus.cc:2781] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:20.747833 8287 raft_consensus.cc:2454] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f58457c81c74bc79e357da433f98539 in term 1.
I20260619 01:07:20.747977 8540 raft_consensus.cc:686] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 1 LEADER]: Becoming Leader. State: Replica: 6f58457c81c74bc79e357da433f98539, State: Running, Role: LEADER
I20260619 01:07:20.748268 8540 consensus_queue.cc:237] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } }
I20260619 01:07:20.751325 7986 catalog_manager.cc:5495] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f58457c81c74bc79e357da433f98539 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6f58457c81c74bc79e357da433f98539" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f58457c81c74bc79e357da433f98539" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42399 } health_report { overall_health: HEALTHY } } }
I20260619 01:07:20.784462 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.778786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.0.97.3:42399)
0619 01:07:20.778967 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:20.778975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:20.779134 (+ 159us) server_negotiation.cc:407] Connection header received
0619 01:07:20.779323 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:20.779327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:20.779395 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:20.779473 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:07:20.779896 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.780592 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.782617 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.783059 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.783164 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.783702 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:07:20.783839 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:07:20.783911 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:20.784078 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 01:07:20.784148 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:07:20.793257 8286 raft_consensus.cc:1261] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Refusing update from remote peer 6f58457c81c74bc79e357da433f98539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:20.793841 8150 raft_consensus.cc:1261] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Refusing update from remote peer 6f58457c81c74bc79e357da433f98539: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:20.794451 8509 consensus_queue.cc:1035] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c6fb42b84ca45d5971e7aa1e69a2c01" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:20.794761 8540 consensus_queue.cc:1035] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f711ef0abb8448eba326bc73972e23a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:20.849622 7986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:20.849875 7986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:20.852952 7986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be08ffb30015417985bb14e2a4360beb: Sending DeleteTablet for 3 replicas of tablet bc9af01195b5477ea67b340ddde2d781
I20260619 01:07:20.853927 8264 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9af01195b5477ea67b340ddde2d781 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45026
I20260619 01:07:20.853989 8405 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9af01195b5477ea67b340ddde2d781 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38078
I20260619 01:07:20.854311 8125 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9af01195b5477ea67b340ddde2d781 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34888
I20260619 01:07:20.854648 8560 tablet_replica.cc:330] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01: stopping tablet replica
I20260619 01:07:20.854826 8560 raft_consensus.cc:2229] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:20.855050 8560 raft_consensus.cc:2258] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:20.855212 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 8059
I20260619 01:07:20.856602 8561 tablet_replica.cc:330] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539: stopping tablet replica
I20260619 01:07:20.856832 8561 raft_consensus.cc:2229] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:07:20.857168 8561 raft_consensus.cc:2258] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:20.857378 8559 tablet_replica.cc:330] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a: stopping tablet replica
I20260619 01:07:20.857599 8559 raft_consensus.cc:2229] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:20.857838 8559 raft_consensus.cc:2258] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:20.858346 8561 ts_tablet_manager.cc:1905] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:20.861017 8560 ts_tablet_manager.cc:1905] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:20.865139 8561 ts_tablet_manager.cc:1918] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:07:20.865473 8561 log.cc:1196] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/bc9af01195b5477ea67b340ddde2d781
I20260619 01:07:20.865998 8561 ts_tablet_manager.cc:1939] T bc9af01195b5477ea67b340ddde2d781 P 6f58457c81c74bc79e357da433f98539: Deleting consensus metadata
I20260619 01:07:20.867208 7973 catalog_manager.cc:4841] TS 6f58457c81c74bc79e357da433f98539 (127.0.97.3:42399): tablet bc9af01195b5477ea67b340ddde2d781 (table test-table [id=23c08d083a4346e9b81cfaefd1e30e8e]) successfully deleted
I20260619 01:07:20.868286 8560 ts_tablet_manager.cc:1918] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:07:20.868458 8560 log.cc:1196] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/bc9af01195b5477ea67b340ddde2d781
I20260619 01:07:20.868836 8560 ts_tablet_manager.cc:1939] T bc9af01195b5477ea67b340ddde2d781 P 9c6fb42b84ca45d5971e7aa1e69a2c01: Deleting consensus metadata
Jun 19 01:07:20 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 01:07:20.870015 7972 catalog_manager.cc:4841] TS 9c6fb42b84ca45d5971e7aa1e69a2c01 (127.0.97.2:38587): tablet bc9af01195b5477ea67b340ddde2d781 (table test-table [id=23c08d083a4346e9b81cfaefd1e30e8e]) successfully deleted
I20260619 01:07:20.870622 8559 ts_tablet_manager.cc:1905] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:20.878726 8559 ts_tablet_manager.cc:1918] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:07:20.878890 8559 log.cc:1196] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/bc9af01195b5477ea67b340ddde2d781
I20260619 01:07:20.879288 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:20.856473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.0.97.1:38745)
0619 01:07:20.856659 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:20.856666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:20.858935 (+ 2269us) server_negotiation.cc:407] Connection header received
0619 01:07:20.859252 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:20.859258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:20.859343 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:20.859422 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:20.861212 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.862458 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.864492 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:20.865251 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:20.865387 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:20.871145 (+ 5758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:20.871166 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:20.871190 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:20.871250 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:20.875500 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:20.876403 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:20.876416 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:20.876419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:20.876673 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:20.877205 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:20.877211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:20.877215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:20.877711 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:20.878046 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:20.878329 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 01:07:20.878600 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 01:07:20.879508 8559 ts_tablet_manager.cc:1939] T bc9af01195b5477ea67b340ddde2d781 P 5f711ef0abb8448eba326bc73972e23a: Deleting consensus metadata
I20260619 01:07:20.880117 8190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:20.880774 7974 catalog_manager.cc:4841] TS 5f711ef0abb8448eba326bc73972e23a (127.0.97.1:38745): tablet bc9af01195b5477ea67b340ddde2d781 (table test-table [id=23c08d083a4346e9b81cfaefd1e30e8e]) successfully deleted
I20260619 01:07:21.003798 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 8059
I20260619 01:07:21.025020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 8199
Jun 19 01:07:21 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 01:07:21.040925 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:21.025856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.0.97.2:38587)
0619 01:07:21.026057 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:21.026063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:21.026083 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:07:21.026199 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:21.026205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:21.026275 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:21.026375 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:21.027579 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:21.028329 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:21.029857 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:21.030268 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:21.030404 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:21.034824 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:21.034846 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:21.034863 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:21.034903 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:21.038001 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:21.038779 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:21.038789 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:21.038793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:21.038951 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:21.039346 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:21.039351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:21.039355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:21.039756 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:21.040050 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:21.040338 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 01:07:21.040579 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:07:21.041661 8324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:21.142179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 8199
I20260619 01:07:21.160207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 8339
Jun 19 01:07:21 dist-test-slave-5p8w krb5kdc[7943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260619 01:07:21.175246 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:21.160981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.0.97.3:42399)
0619 01:07:21.161167 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:21.161171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:21.161191 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:07:21.161407 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:21.161410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:21.161475 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:21.161563 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:21.162800 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:21.163530 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:21.164739 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:21.165140 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:21.165250 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:21.169675 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:21.169691 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:21.169707 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:21.169753 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:21.172486 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:21.173186 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:21.173194 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:21.173196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:21.173489 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:21.173841 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:21.173844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:21.173846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:21.174235 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:21.174461 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:21.174696 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 01:07:21.174888 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:07:21.176033 8470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:21.321334 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 8339
I20260619 01:07:21.346866 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 7956
I20260619 01:07:21.348100 8028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:21.464980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 7956
2026-06-19T01:07:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5923 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:07:21 dist-test-slave-5p8w krb5kdc[8605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:07:21 dist-test-slave-5p8w krb5kdc[8605](info): set up 2 sockets
Jun 19 01:07:21 dist-test-slave-5p8w krb5kdc[8605](info): commencing operation
krb5kdc: starting...
W20260619 01:07:22.562286 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:07:22 dist-test-slave-5p8w krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:07:22.795164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37733
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:23.139433 8618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:23.139746 8618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:23.139914 8618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:23.147456 8618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:07:23.147583 8618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:23.147630 8618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:23.147703 8618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:07:23.147782 8618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:07:23.158388 8618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:23.159842 8618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:23.161347 8618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:23.174263 8623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:23.174222 8624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:23.174312 8626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:23.175451 8618 server_base.cc:1034] running on GCE node
I20260619 01:07:23.176396 8618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:23.178052 8618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:23.179307 8618 hybrid_clock.cc:648] HybridClock initialized: now 1781831243179259 us; error 61 us; skew 500 ppm
I20260619 01:07:23.179688 8618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T01:07:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1777 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:07:23 dist-test-slave-5p8w krb5kdc[8632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:07:23 dist-test-slave-5p8w krb5kdc[8632](info): set up 2 sockets
Jun 19 01:07:23 dist-test-slave-5p8w krb5kdc[8632](info): commencing operation
krb5kdc: starting...
W20260619 01:07:24.346289 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:07:24 dist-test-slave-5p8w krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:07:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:07:24.583093 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40369
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:24.932385 8645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:24.932734 8645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:24.932888 8645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:24.940327 8645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:07:24.940455 8645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:24.940536 8645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:24.940618 8645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:07:24.940706 8645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:07:24.951092 8645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:24.952489 8645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:24.954079 8645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:24.967365 8651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:24.967401 8650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:24.967737 8653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:24.968405 8645 server_base.cc:1034] running on GCE node
I20260619 01:07:24.969383 8645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:24.970916 8645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:24.972147 8645 hybrid_clock.cc:648] HybridClock initialized: now 1781831244972078 us; error 60 us; skew 500 ppm
I20260619 01:07:24.972566 8645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:24 dist-test-slave-5p8w krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:24.980913 8645 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:07:24.982748 8645 webserver.cc:460] Webserver started at http://127.0.97.62:35879/ using document root <none> and password file <none>
I20260619 01:07:24.983543 8645 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:24.983682 8645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:24.984025 8645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:24.987260 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "11c7f02c3ea4492883a56987822d7cbd"
format_stamp: "Formatted at 2026-06-19 01:07:24 on dist-test-slave-5p8w"
server_key: "29f44a9ab9677a7fa169549b3e0ede19"
server_key_iv: "f3cc28de997966a098c462180567c7cb"
server_key_version: "encryptionkey@0"
I20260619 01:07:24.988083 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "11c7f02c3ea4492883a56987822d7cbd"
format_stamp: "Formatted at 2026-06-19 01:07:24 on dist-test-slave-5p8w"
server_key: "29f44a9ab9677a7fa169549b3e0ede19"
server_key_iv: "f3cc28de997966a098c462180567c7cb"
server_key_version: "encryptionkey@0"
I20260619 01:07:24.993607 8645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 01:07:24.997376 8660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:24.998803 8645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:24.998962 8645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "11c7f02c3ea4492883a56987822d7cbd"
format_stamp: "Formatted at 2026-06-19 01:07:24 on dist-test-slave-5p8w"
server_key: "29f44a9ab9677a7fa169549b3e0ede19"
server_key_iv: "f3cc28de997966a098c462180567c7cb"
server_key_version: "encryptionkey@0"
I20260619 01:07:24.999136 8645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:25.019906 8645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:25.024529 8645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:25.024771 8645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:25.044302 8645 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40369
I20260619 01:07:25.044324 8711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40369 every 8 connection(s)
I20260619 01:07:25.045823 8645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:07:25.046864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 8645
I20260619 01:07:25.047113 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:07:25.047722 388 external_mini_cluster.cc:1428] Setting key 03de60b0934d50558b437eb11424f433
I20260619 01:07:25.050542 8712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:07:25 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:25.063997 8712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd: Bootstrap starting.
I20260619 01:07:25.068647 8712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:25.070097 8712 log.cc:824] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:25.073213 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:25.049754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.0.97.62:40369)
0619 01:07:25.050590 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:25.050607 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:25.050661 (+ 54us) server_negotiation.cc:407] Connection header received
0619 01:07:25.052935 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:25.052965 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:25.053535 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:25.054038 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:25.055357 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:25.057538 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:25.058908 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:25.059980 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:25.060189 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:25.064137 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:25.064151 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:25.064174 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:25.064224 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:25.068239 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:25.069094 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:25.069110 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:25.069122 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:25.069382 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:25.069762 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:25.069767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:25.069770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:25.070513 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:25.070904 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:25.071339 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 01:07:25.071779 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":329,"threads_started":1}
I20260619 01:07:25.073850 8712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd: No bootstrap required, opened a new log
I20260619 01:07:25.082348 8712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } }
I20260619 01:07:25.082765 8712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:25.082878 8712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c7f02c3ea4492883a56987822d7cbd, State: Initialized, Role: FOLLOWER
I20260619 01:07:25.083513 8712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } }
I20260619 01:07:25.083753 8712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:07:25.083900 8712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:07:25.084085 8712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:25.086388 8712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } }
I20260619 01:07:25.086959 8712 leader_election.cc:302] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11c7f02c3ea4492883a56987822d7cbd; no voters:
I20260619 01:07:25.087566 8712 leader_election.cc:288] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:07:25.087718 8717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:25.088480 8717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 1 LEADER]: Becoming Leader. State: Replica: 11c7f02c3ea4492883a56987822d7cbd, State: Running, Role: LEADER
I20260619 01:07:25.089001 8717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } }
I20260619 01:07:25.089874 8712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:07:25.092232 8719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11c7f02c3ea4492883a56987822d7cbd. Latest consensus state: current_term: 1 leader_uuid: "11c7f02c3ea4492883a56987822d7cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } } }
I20260619 01:07:25.092682 8719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:25.093043 8718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11c7f02c3ea4492883a56987822d7cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } } }
I20260619 01:07:25.093510 8718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:25.094542 8724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:07:25.099970 8724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:07:25.109205 8724 catalog_manager.cc:1347] Generated new cluster ID: 58b8f441bae64d228669c03d58f7ea96
I20260619 01:07:25.109359 8724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:07:25.128266 8724 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:07:25.130719 8724 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:07:25.145910 8724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd: Generated new TSK 0
I20260619 01:07:25.146832 8724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:07:25.217551 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40369
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:25.563479 8738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:25.563817 8738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:25.563977 8738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:25.571529 8738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:25.571712 8738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:25.571939 8738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:07:25.582526 8738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:25.583817 8738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:25.585127 8738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:25.598971 8744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:25.599016 8743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:25.599691 8746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:25.599819 8738 server_base.cc:1034] running on GCE node
I20260619 01:07:25.600490 8738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:25.601830 8738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:25.603077 8738 hybrid_clock.cc:648] HybridClock initialized: now 1781831245603022 us; error 76 us; skew 500 ppm
I20260619 01:07:25.603434 8738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:25 dist-test-slave-5p8w krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:25.606703 8738 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:07:25.608142 8738 webserver.cc:460] Webserver started at http://127.0.97.1:34561/ using document root <none> and password file <none>
I20260619 01:07:25.608840 8738 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:25.608947 8738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:25.609205 8738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:25.612396 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "6cef919a42fe45a49669cfbe28046ef4"
format_stamp: "Formatted at 2026-06-19 01:07:25 on dist-test-slave-5p8w"
server_key: "307fc2be33bdb47cd18ae0e5d9c3d177"
server_key_iv: "728c19fe4abac7fcd2fe41c6b3b6c657"
server_key_version: "encryptionkey@0"
I20260619 01:07:25.613139 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6cef919a42fe45a49669cfbe28046ef4"
format_stamp: "Formatted at 2026-06-19 01:07:25 on dist-test-slave-5p8w"
server_key: "307fc2be33bdb47cd18ae0e5d9c3d177"
server_key_iv: "728c19fe4abac7fcd2fe41c6b3b6c657"
server_key_version: "encryptionkey@0"
I20260619 01:07:25.618417 8738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 01:07:25.622027 8753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:25.623504 8738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:25.623708 8738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "6cef919a42fe45a49669cfbe28046ef4"
format_stamp: "Formatted at 2026-06-19 01:07:25 on dist-test-slave-5p8w"
server_key: "307fc2be33bdb47cd18ae0e5d9c3d177"
server_key_iv: "728c19fe4abac7fcd2fe41c6b3b6c657"
server_key_version: "encryptionkey@0"
I20260619 01:07:25.623930 8738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:25.642977 8738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:25.647262 8738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:25.647567 8738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:25.648571 8738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:25.650022 8738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:25.650096 8738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:25.650265 8738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:25.650322 8738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:25.683280 8738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36839
I20260619 01:07:25.683320 8865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36839 every 8 connection(s)
I20260619 01:07:25.684607 8738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:07:25.693356 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 8738
I20260619 01:07:25.693598 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:07:25.694375 388 external_mini_cluster.cc:1428] Setting key 1a55e89419979e56fba0cacff3e9fb5d
Jun 19 01:07:25 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:25.706960 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:25.687285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35941 (local address 127.0.97.62:40369)
0619 01:07:25.687758 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:25.687765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:25.688794 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 01:07:25.690776 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:25.690781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:25.690856 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:25.691029 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:25.692552 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:25.693260 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:25.695508 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:25.695960 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:25.696117 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:25.700443 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:25.700450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:25.700453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:25.700491 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:25.703215 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:25.704265 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:25.704272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:25.704277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:25.704454 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:25.704883 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:25.704891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:25.704898 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:25.705282 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:25.705504 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:25.706193 (+ 689us) server_negotiation.cc:299] Negotiation successful
0619 01:07:25.706470 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":227,"threads_started":1}
I20260619 01:07:25.707701 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:25.687917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:35941)
0619 01:07:25.688632 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:25.688670 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:25.690477 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:25.691225 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:25.691235 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:25.691787 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:25.692383 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:25.692401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:25.693481 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:25.693487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:25.695378 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:25.695391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:25.696213 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:25.696223 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:25.696493 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:25.697261 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:25.697345 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:25.700144 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:25.703481 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:25.703490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:25.703506 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:25.704068 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:25.704569 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:25.704573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:25.704575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:25.704776 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:25.705447 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:25.705453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:25.705815 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 01:07:25.706063 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 01:07:25.706509 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":273,"threads_started":1}
I20260619 01:07:25.709408 8866 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40369
I20260619 01:07:25.709771 8866 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:25.710733 8866 heartbeater.cc:511] Master 127.0.97.62:40369 requested a full tablet report, sending...
I20260619 01:07:25.713155 8677 ts_manager.cc:194] Registered new tserver with Master: 6cef919a42fe45a49669cfbe28046ef4 (127.0.97.1:36839)
I20260619 01:07:25.714713 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35941
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:07:25.748238 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40369
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:26.116235 8872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:26.116695 8872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:26.116882 8872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:26.133335 8872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:26.133517 8872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:26.133771 8872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:07:26.155014 8872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:26.156906 8872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:26.158906 8872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:26.177069 8878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:26.177474 8877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:26.177505 8880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:26.178395 8872 server_base.cc:1034] running on GCE node
I20260619 01:07:26.179347 8872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:26.180855 8872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:26.182152 8872 hybrid_clock.cc:648] HybridClock initialized: now 1781831246182057 us; error 73 us; skew 500 ppm
I20260619 01:07:26.182718 8872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:26 dist-test-slave-5p8w krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:26.186607 8872 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:07:26.190002 8872 webserver.cc:460] Webserver started at http://127.0.97.2:34065/ using document root <none> and password file <none>
I20260619 01:07:26.190800 8872 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:26.190950 8872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:26.191303 8872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:26.194398 8872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "7cf38ef02cc84fa9ad766c3aeb17eb07"
format_stamp: "Formatted at 2026-06-19 01:07:26 on dist-test-slave-5p8w"
server_key: "f1aeea7db50abaca16b8dd61706520e7"
server_key_iv: "0bf1ca1980e09e35e0fc44fe7bdfd9db"
server_key_version: "encryptionkey@0"
I20260619 01:07:26.195276 8872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7cf38ef02cc84fa9ad766c3aeb17eb07"
format_stamp: "Formatted at 2026-06-19 01:07:26 on dist-test-slave-5p8w"
server_key: "f1aeea7db50abaca16b8dd61706520e7"
server_key_iv: "0bf1ca1980e09e35e0fc44fe7bdfd9db"
server_key_version: "encryptionkey@0"
I20260619 01:07:26.201510 8872 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260619 01:07:26.205924 8887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:26.207660 8872 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 01:07:26.207892 8872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "7cf38ef02cc84fa9ad766c3aeb17eb07"
format_stamp: "Formatted at 2026-06-19 01:07:26 on dist-test-slave-5p8w"
server_key: "f1aeea7db50abaca16b8dd61706520e7"
server_key_iv: "0bf1ca1980e09e35e0fc44fe7bdfd9db"
server_key_version: "encryptionkey@0"
I20260619 01:07:26.208127 8872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:26.265949 8872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:26.270789 8872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:26.271095 8872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:26.272207 8872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:26.273790 8872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:26.273871 8872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:26.273998 8872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:26.274048 8872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:26.314306 8872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36863
I20260619 01:07:26.314327 8999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36863 every 8 connection(s)
I20260619 01:07:26.315658 8872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:07:26.323496 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 8872
I20260619 01:07:26.323709 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:07:26.324216 388 external_mini_cluster.cc:1428] Setting key db84c0579f2090e03c92f74b5a4f0acd
Jun 19 01:07:26 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:26.343645 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:26.318622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60397 (local address 127.0.97.62:40369)
0619 01:07:26.319090 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:26.319095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:26.320479 (+ 1384us) server_negotiation.cc:407] Connection header received
0619 01:07:26.322431 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:26.322437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:26.322528 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:26.322654 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:26.324818 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:26.326209 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:26.328886 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:26.329777 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:26.329935 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:26.335170 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:26.335180 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:26.335188 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:26.335239 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:26.339120 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:26.340285 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:26.340293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:26.340299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:26.340457 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:26.341044 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:26.341052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:26.341058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:26.341494 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:26.341660 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:26.342825 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0619 01:07:26.343108 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":256,"threads_started":1}
I20260619 01:07:26.345166 9003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:26.319453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:60397)
0619 01:07:26.320306 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:26.320347 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:26.322140 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:26.322916 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:26.322925 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:26.323705 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:26.324583 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:26.324610 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:26.326388 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:26.326394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:26.328702 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:26.328722 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:26.330057 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:26.330067 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:26.330420 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:26.331304 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:26.331339 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:26.334861 (+ 3522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:26.339319 (+ 4458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:26.339334 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:26.339359 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:26.340129 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:26.340588 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:26.340604 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:26.340609 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:26.340907 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:26.341708 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:26.341720 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:26.342314 (+ 594us) client_negotiation.cc:769] Sending connection context
0619 01:07:26.342726 (+ 412us) client_negotiation.cc:240] Negotiation successful
0619 01:07:26.343457 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":350,"threads_started":1}
I20260619 01:07:26.347062 9000 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40369
I20260619 01:07:26.347466 9000 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:26.348500 9000 heartbeater.cc:511] Master 127.0.97.62:40369 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:07:26.350824 8677 ts_manager.cc:194] Registered new tserver with Master: 7cf38ef02cc84fa9ad766c3aeb17eb07 (127.0.97.2:36863)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 01:07:26.352211 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:07:26.380798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40369
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:07:26.718837 8866 heartbeater.cc:503] Master 127.0.97.62:40369 was elected leader, sending a full tablet report...
W20260619 01:07:26.723582 9006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:26.723881 9006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:26.724059 9006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:26.731706 9006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:26.731873 9006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:26.732067 9006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:07:26.742133 9006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:26.743505 9006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:26.744982 9006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:26.758781 9011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:26.758857 9012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:26.758857 9014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:26.760006 9006 server_base.cc:1034] running on GCE node
I20260619 01:07:26.760710 9006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:26.761878 9006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:26.763114 9006 hybrid_clock.cc:648] HybridClock initialized: now 1781831246763046 us; error 69 us; skew 500 ppm
I20260619 01:07:26.763468 9006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:26 dist-test-slave-5p8w krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:26.766916 9006 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:07:26.768452 9006 webserver.cc:460] Webserver started at http://127.0.97.3:45283/ using document root <none> and password file <none>
I20260619 01:07:26.769164 9006 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:26.769270 9006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:26.769624 9006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:26.772809 9006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "4a2bbed747fc4127b459835842cf3b1f"
format_stamp: "Formatted at 2026-06-19 01:07:26 on dist-test-slave-5p8w"
server_key: "140e9ca9b9ef630ceb6f3adc9db11ff1"
server_key_iv: "89d2fa07df163e6b2f63cc5d0753b670"
server_key_version: "encryptionkey@0"
I20260619 01:07:26.773722 9006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4a2bbed747fc4127b459835842cf3b1f"
format_stamp: "Formatted at 2026-06-19 01:07:26 on dist-test-slave-5p8w"
server_key: "140e9ca9b9ef630ceb6f3adc9db11ff1"
server_key_iv: "89d2fa07df163e6b2f63cc5d0753b670"
server_key_version: "encryptionkey@0"
I20260619 01:07:26.779364 9006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 01:07:26.783386 9021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:26.784933 9006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:07:26.785147 9006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "4a2bbed747fc4127b459835842cf3b1f"
format_stamp: "Formatted at 2026-06-19 01:07:26 on dist-test-slave-5p8w"
server_key: "140e9ca9b9ef630ceb6f3adc9db11ff1"
server_key_iv: "89d2fa07df163e6b2f63cc5d0753b670"
server_key_version: "encryptionkey@0"
I20260619 01:07:26.785409 9006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:26.810665 9006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:26.815618 9006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:26.815886 9006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:26.816854 9006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:26.818488 9006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:26.818569 9006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:26.818660 9006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:26.818706 9006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:26.853518 9006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44021
I20260619 01:07:26.853565 9133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44021 every 8 connection(s)
I20260619 01:07:26.854903 9006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:07:26.863543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 9006
I20260619 01:07:26.863756 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:07:26.864364 388 external_mini_cluster.cc:1428] Setting key 3e24b68393c54926c14510f6b79b35db
Jun 19 01:07:26 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:26.875764 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:26.857550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60987 (local address 127.0.97.62:40369)
0619 01:07:26.857966 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:26.857971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:26.859139 (+ 1168us) server_negotiation.cc:407] Connection header received
0619 01:07:26.861000 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:26.861005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:26.861074 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:26.861175 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:26.862833 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:26.863616 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:26.865885 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:26.866295 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:26.866444 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:26.870214 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:26.870219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:26.870222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:26.870268 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:26.872863 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:26.873742 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:26.873746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:26.873748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:26.873881 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:26.874266 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:26.874269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:26.874271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:26.874525 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:26.874631 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:26.875321 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 01:07:26.875490 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":212,"threads_started":1}
I20260619 01:07:26.876847 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:26.858209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:60987)
0619 01:07:26.858955 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:26.859005 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:26.860695 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:26.861471 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:26.861479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:26.862032 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:26.862643 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:26.862659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:26.863827 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:26.863835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:26.865740 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:26.865753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:26.866531 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:26.866540 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:26.866813 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:26.867606 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:26.867637 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:26.869964 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:26.873029 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:26.873037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:26.873053 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:26.873622 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:26.873974 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:26.873977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:26.873980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:26.874171 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:26.874631 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:26.874638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:26.874986 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 01:07:26.875228 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 01:07:26.875646 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":282,"threads_started":1}
I20260619 01:07:26.878417 9134 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40369
I20260619 01:07:26.878738 9134 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:26.879477 9134 heartbeater.cc:511] Master 127.0.97.62:40369 requested a full tablet report, sending...
I20260619 01:07:26.880885 8677 ts_manager.cc:194] Registered new tserver with Master: 4a2bbed747fc4127b459835842cf3b1f (127.0.97.3:44021)
I20260619 01:07:26.881742 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60987
I20260619 01:07:26.892277 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 01:07:27.037146 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 8738
Jun 19 01:07:27 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:07:27.054665 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:27.038365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.0.97.1:36839)
0619 01:07:27.038936 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:27.038944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:27.038978 (+ 34us) server_negotiation.cc:407] Connection header received
0619 01:07:27.039176 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:27.039183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:27.039407 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:27.039553 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:27.041036 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:27.042146 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:27.043437 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:27.044124 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:27.044273 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:27.048695 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:27.048708 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:27.048723 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:27.048768 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:27.051748 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:27.052486 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:27.052500 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:27.052502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:27.052626 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:27.052996 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:27.053004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:27.053007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:27.053449 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:27.053771 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:27.054024 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 01:07:27.054310 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":267,"threads_started":1}
I20260619 01:07:27.055956 8861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:27.141873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 8738
I20260619 01:07:27.158456 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 8872
Jun 19 01:07:27 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:07:27.174817 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:27.159392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.0.97.2:36863)
0619 01:07:27.159881 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:27.159888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:27.159912 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:07:27.159969 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:27.159975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:27.160183 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:27.160312 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:27.161617 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:27.162630 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:27.163839 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:27.164364 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:27.164459 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:27.168649 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:27.168662 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:27.168678 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:27.168716 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:27.172007 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:27.172832 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:27.172840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:27.172842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:27.172955 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:27.173250 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:27.173253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:27.173254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:27.173753 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:27.173992 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:27.174209 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 01:07:27.174405 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":244,"threads_started":1}
I20260619 01:07:27.176118 8995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:27.260075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 8872
I20260619 01:07:27.276286 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 9006
Jun 19 01:07:27 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:07:27.293419 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:27.277271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.0.97.3:44021)
0619 01:07:27.277999 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:27.278006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:27.278034 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:07:27.278089 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:27.278096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:27.278308 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:27.278437 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:27.279731 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:27.280760 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:27.282315 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:27.282970 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:27.283070 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:27.287321 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:27.287335 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:27.287351 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:27.287395 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:27.290257 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:27.291111 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:27.291118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:27.291121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:27.291258 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:27.291740 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:27.291745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:27.291747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:27.292111 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:27.292451 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:27.292701 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 01:07:27.292977 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":330,"threads_started":1}
I20260619 01:07:27.294708 9129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:27.378281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 9006
I20260619 01:07:27.394305 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 8645
I20260619 01:07:27.395460 8707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:27.499109 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 8645
I20260619 01:07:27.520179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40369
--webserver_interface=127.0.97.62
--webserver_port=35879
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:27.849617 9152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:27.849988 9152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:27.850169 9152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:27.857999 9152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:07:27.858196 9152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:27.858304 9152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:27.858405 9152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:07:27.858484 9152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:07:27.869221 9152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:27.870669 9152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:27.872023 9152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:27.884078 9157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:27.884091 9160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:27.884123 9158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:27.884796 9152 server_base.cc:1034] running on GCE node
I20260619 01:07:27.885703 9152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:27.887153 9152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:27.888391 9152 hybrid_clock.cc:648] HybridClock initialized: now 1781831247888348 us; error 55 us; skew 500 ppm
I20260619 01:07:27.888751 9152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:27 dist-test-slave-5p8w krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:27.891724 9152 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:07:27.893282 9152 webserver.cc:460] Webserver started at http://127.0.97.62:35879/ using document root <none> and password file <none>
I20260619 01:07:27.894011 9152 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:27.894161 9152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:27.899616 9152 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260619 01:07:27.902700 9167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:27.904078 9152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:27.904280 9152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "11c7f02c3ea4492883a56987822d7cbd"
format_stamp: "Formatted at 2026-06-19 01:07:24 on dist-test-slave-5p8w"
server_key: "29f44a9ab9677a7fa169549b3e0ede19"
server_key_iv: "f3cc28de997966a098c462180567c7cb"
server_key_version: "encryptionkey@0"
I20260619 01:07:27.905288 9152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:27.924132 9152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:27.928378 9152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:27.928680 9152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:27.947837 9152 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40369
I20260619 01:07:27.947858 9218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40369 every 8 connection(s)
I20260619 01:07:27.949512 9152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:07:27.952147 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 9152
I20260619 01:07:27.953083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36839
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40369
--webserver_port=34561
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:07:27.959205 9219 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:07:27.963449 9219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd: Bootstrap starting.
I20260619 01:07:27.983718 9219 log.cc:824] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:27.992588 9219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:07:27.993141 9219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd: Bootstrap complete.
I20260619 01:07:28.002362 9219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } }
I20260619 01:07:28.003216 9219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c7f02c3ea4492883a56987822d7cbd, State: Initialized, Role: FOLLOWER
I20260619 01:07:28.003861 9219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } }
I20260619 01:07:28.004093 9219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:07:28.004215 9219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:07:28.004406 9219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:07:28.008287 9219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } }
I20260619 01:07:28.008853 9219 leader_election.cc:302] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11c7f02c3ea4492883a56987822d7cbd; no voters:
I20260619 01:07:28.009549 9219 leader_election.cc:288] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 01:07:28.009711 9223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 2 FOLLOWER]: Leader election won for term 2
I20260619 01:07:28.011889 9219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:07:28.017014 9223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [term 2 LEADER]: Becoming Leader. State: Replica: 11c7f02c3ea4492883a56987822d7cbd, State: Running, Role: LEADER
I20260619 01:07:28.017834 9223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } }
I20260619 01:07:28.020764 9224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "11c7f02c3ea4492883a56987822d7cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } } }
I20260619 01:07:28.021219 9224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:28.021867 9225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11c7f02c3ea4492883a56987822d7cbd. Latest consensus state: current_term: 2 leader_uuid: "11c7f02c3ea4492883a56987822d7cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c7f02c3ea4492883a56987822d7cbd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40369 } } }
I20260619 01:07:28.022158 9225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:28.022161 9234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:07:28.034050 9234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:07:28.035979 9234 catalog_manager.cc:1259] Loaded cluster ID: 58b8f441bae64d228669c03d58f7ea96
I20260619 01:07:28.036227 9234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:07:28.041698 9234 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:07:28.043874 9234 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 11c7f02c3ea4492883a56987822d7cbd: Loaded TSK: 0
I20260619 01:07:28.046064 9234 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 01:07:28.328315 9221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:28.328603 9221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:28.328744 9221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:28.336161 9221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:28.336284 9221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:28.336452 9221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:07:28.346544 9221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34561
--tserver_master_addrs=127.0.97.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:28.347904 9221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:28.349195 9221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:28.362628 9246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:28.362628 9247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:28.362820 9249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:28.364282 9221 server_base.cc:1034] running on GCE node
I20260619 01:07:28.365010 9221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:28.366181 9221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:28.367419 9221 hybrid_clock.cc:648] HybridClock initialized: now 1781831248367371 us; error 43 us; skew 500 ppm
I20260619 01:07:28.367762 9221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831248, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:28.371210 9221 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 01:07:28.372879 9221 webserver.cc:460] Webserver started at http://127.0.97.1:34561/ using document root <none> and password file <none>
I20260619 01:07:28.373667 9221 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:28.373816 9221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:28.379593 9221 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:07:28.382504 9256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:28.383963 9221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:28.384136 9221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "6cef919a42fe45a49669cfbe28046ef4"
format_stamp: "Formatted at 2026-06-19 01:07:25 on dist-test-slave-5p8w"
server_key: "307fc2be33bdb47cd18ae0e5d9c3d177"
server_key_iv: "728c19fe4abac7fcd2fe41c6b3b6c657"
server_key_version: "encryptionkey@0"
I20260619 01:07:28.385084 9221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:28.403298 9221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:28.407518 9221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:28.407780 9221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:28.408779 9221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:28.410219 9221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:28.410328 9221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:28.410454 9221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:28.410542 9221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:28.443346 9221 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36839
I20260619 01:07:28.443411 9368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36839 every 8 connection(s)
I20260619 01:07:28.444873 9221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:07:28.450424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 9221
I20260619 01:07:28.451134 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36863
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40369
--webserver_port=34065
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 01:07:28.467057 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:28.447797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51419 (local address 127.0.97.62:40369)
0619 01:07:28.448561 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:28.448574 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:28.448924 (+ 350us) server_negotiation.cc:407] Connection header received
0619 01:07:28.451722 (+ 2798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:28.451736 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:28.452095 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:28.452411 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:28.454483 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.456227 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.458552 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.459719 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.459937 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.466135 (+ 6198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51419: BlockingRecv error: recv got EOF from 127.0.97.1:51419 (error 108)
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":291,"threads_started":1}
I20260619 01:07:28.467132 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:28.447926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:51419)
0619 01:07:28.448732 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:28.448785 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:28.451106 (+ 2321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:28.452613 (+ 1507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:28.452627 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:28.453464 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:28.454254 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.454280 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.456407 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.456416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.458406 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.458419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.459937 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.459942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.460146 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.460843 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:28.460867 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:28.465804 (+ 4937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":358,"threads_started":1}
W20260619 01:07:28.468019 9369 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40369 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:40369: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:28.478557 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:28.468636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:56351)
0619 01:07:28.468850 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:28.468882 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:28.469018 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:28.469426 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:28.469433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:28.469946 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:28.470181 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.470197 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.471148 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.471157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.472929 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.472949 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.473653 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.473662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.473902 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.474528 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:28.474556 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:28.478274 (+ 3718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 01:07:28.478612 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:28.468685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56351 (local address 127.0.97.62:40369)
0619 01:07:28.468846 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:28.468850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:28.468957 (+ 107us) server_negotiation.cc:407] Connection header received
0619 01:07:28.469160 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:28.469167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:28.469250 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:28.469384 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:28.470322 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.471002 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.473070 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.473506 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.473596 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.478446 (+ 4850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56351: BlockingRecv error: recv got EOF from 127.0.97.1:56351 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:28.488736 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:28.479564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:48219)
0619 01:07:28.479742 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:28.479760 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:28.479860 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:28.480206 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:28.480211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:28.480580 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:28.480823 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.480834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.481906 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.481912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.483419 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.483433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.484150 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.484159 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.484382 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.484995 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:28.485018 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:28.488382 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 01:07:28.488778 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:28.479761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48219 (local address 127.0.97.62:40369)
0619 01:07:28.479921 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:28.479925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:28.479943 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:07:28.479992 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:28.479995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:28.480068 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:28.480155 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:28.480981 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.481771 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.483569 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.483996 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.484085 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.488576 (+ 4491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48219: BlockingRecv error: recv got EOF from 127.0.97.1:48219 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 01:07:28.489198 9369 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:07:28.803895 9373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:28.804199 9373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:28.804412 9373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:28.812464 9373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:28.812628 9373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:28.812832 9373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:07:28.823797 9373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34065
--tserver_master_addrs=127.0.97.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:28.825279 9373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:28.826676 9373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:28.840446 9381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:28.840458 9379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:28.840775 9378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:28.841338 9373 server_base.cc:1034] running on GCE node
I20260619 01:07:28.842011 9373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:28.842971 9373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:28.844168 9373 hybrid_clock.cc:648] HybridClock initialized: now 1781831248844100 us; error 64 us; skew 500 ppm
I20260619 01:07:28.844480 9373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831248, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:28.847785 9373 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 01:07:28.849449 9373 webserver.cc:460] Webserver started at http://127.0.97.2:34065/ using document root <none> and password file <none>
I20260619 01:07:28.850205 9373 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:28.850333 9373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:28.856303 9373 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 01:07:28.859153 9388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:28.860620 9373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:07:28.860807 9373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "7cf38ef02cc84fa9ad766c3aeb17eb07"
format_stamp: "Formatted at 2026-06-19 01:07:26 on dist-test-slave-5p8w"
server_key: "f1aeea7db50abaca16b8dd61706520e7"
server_key_iv: "0bf1ca1980e09e35e0fc44fe7bdfd9db"
server_key_version: "encryptionkey@0"
I20260619 01:07:28.861892 9373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:28.891305 9373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:28.896445 9373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:28.896777 9373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:28.897964 9373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:28.899453 9373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:28.899601 9373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:28.899758 9373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:28.899857 9373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:28.936491 9373 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36863
I20260619 01:07:28.936590 9500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36863 every 8 connection(s)
I20260619 01:07:28.938340 9373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:07:28.948344 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 9373
I20260619 01:07:28.949169 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44021
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40369
--webserver_port=45283
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 01:07:28.955173 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:28.941276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53133 (local address 127.0.97.62:40369)
0619 01:07:28.941479 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:28.941484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:28.942819 (+ 1335us) server_negotiation.cc:407] Connection header received
0619 01:07:28.944837 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:28.944843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:28.944937 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:28.945050 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:28.946812 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.947563 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.949363 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.949902 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.950054 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.954922 (+ 4868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53133: BlockingRecv error: recv got EOF from 127.0.97.2:53133 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 01:07:28.956050 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:28.941828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:53133)
0619 01:07:28.942616 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:28.942661 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:28.944520 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:28.945368 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:28.945377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:28.945995 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:28.946620 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.946637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.947748 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.947752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.949173 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.949186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.950149 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.950156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.950403 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.951119 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:28.951143 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:28.954749 (+ 3606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":326,"threads_started":1}
W20260619 01:07:28.956727 9501 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40369 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:40369: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:28.965173 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:28.957396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:59243)
0619 01:07:28.957646 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:28.957664 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:28.957794 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:28.958128 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:28.958132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:28.958471 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:28.958613 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.958621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.959634 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.959639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.960533 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.960544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.961265 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.961271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.961483 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.962082 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:28.962100 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:28.964852 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138}
W20260619 01:07:28.965355 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:28.957568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59243 (local address 127.0.97.62:40369)
0619 01:07:28.957765 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:28.957773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:28.957796 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:07:28.957925 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:28.957930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:28.958001 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:28.958104 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:28.958753 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.959497 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.960671 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.961104 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.961263 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.965068 (+ 3805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59243: BlockingRecv error: recv got EOF from 127.0.97.2:59243 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:28 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:28.974754 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:28.966170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:55913)
0619 01:07:28.966465 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:28.966482 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:28.966622 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:28.966979 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:28.966983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:28.967311 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:28.967456 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.967465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.968475 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.968480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.969598 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:28.969611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.970344 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.970350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:28.970527 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.971125 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:28.971142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:28.974436 (+ 3294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189}
W20260619 01:07:28.974885 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:28.966244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55913 (local address 127.0.97.62:40369)
0619 01:07:28.966411 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:28.966417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:28.966561 (+ 144us) server_negotiation.cc:407] Connection header received
0619 01:07:28.966792 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:28.966796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:28.966853 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:28.966952 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:28.967591 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.968341 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.969749 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:28.970198 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:28.970329 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:28.974651 (+ 4322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55913: BlockingRecv error: recv got EOF from 127.0.97.2:55913 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 01:07:28.975119 9501 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:07:29.296991 9504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:29.297394 9504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:29.297573 9504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:29.305009 9504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:29.305133 9504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:29.305415 9504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:07:29.315905 9504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45283
--tserver_master_addrs=127.0.97.62:40369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:29.317396 9504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:29.318894 9504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:29.333091 9509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:29.333055 9510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:29.333124 9512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:29.334549 9504 server_base.cc:1034] running on GCE node
I20260619 01:07:29.335255 9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:29.336299 9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:29.337558 9504 hybrid_clock.cc:648] HybridClock initialized: now 1781831249337494 us; error 71 us; skew 500 ppm
I20260619 01:07:29.337894 9504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:29 dist-test-slave-5p8w krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831249, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:29.341151 9504 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 01:07:29.342793 9504 webserver.cc:460] Webserver started at http://127.0.97.3:45283/ using document root <none> and password file <none>
I20260619 01:07:29.343557 9504 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:29.343708 9504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:29.349798 9504 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 01:07:29.352953 9519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:29.354723 9504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:29.354943 9504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "4a2bbed747fc4127b459835842cf3b1f"
format_stamp: "Formatted at 2026-06-19 01:07:26 on dist-test-slave-5p8w"
server_key: "140e9ca9b9ef630ceb6f3adc9db11ff1"
server_key_iv: "89d2fa07df163e6b2f63cc5d0753b670"
server_key_version: "encryptionkey@0"
I20260619 01:07:29.356031 9504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:29.375425 9504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:29.380030 9504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:29.380317 9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:29.381417 9504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:29.382797 9504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:29.382931 9504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:29.383061 9504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:29.383152 9504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:29.417474 9504 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44021
I20260619 01:07:29.417519 9631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44021 every 8 connection(s)
I20260619 01:07:29.419194 9504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:07:29.427436 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 9504
Jun 19 01:07:29 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:29 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 01:07:29.437512 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:29.422047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41317 (local address 127.0.97.62:40369)
0619 01:07:29.422257 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:29.422263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:29.423599 (+ 1336us) server_negotiation.cc:407] Connection header received
0619 01:07:29.425711 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:29.425717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:29.425820 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:29.425937 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:29.427949 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.429158 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.431183 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.431679 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.431819 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:29.437156 (+ 5337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41317: BlockingRecv error: recv got EOF from 127.0.97.3:41317 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":74}
I20260619 01:07:29.438261 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:29.422611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:41317)
0619 01:07:29.423407 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:29.423448 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:29.425384 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:29.426227 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:29.426237 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:29.426803 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:29.427717 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:29.427746 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.429422 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.429428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:29.431014 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:29.431029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.431921 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.431929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:29.432185 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:29.432929 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:29.432952 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:29.436885 (+ 3933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":333,"threads_started":1}
W20260619 01:07:29.439113 9632 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40369 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:40369: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260619 01:07:29.445281 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:29.428602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.0.97.62:40369)
0619 01:07:29.429427 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:29.429432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:29.429448 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:07:29.429606 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:29.429611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:29.429686 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:29.429797 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:29.431358 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.432226 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.433925 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.434660 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.434799 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:29.437522 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:29.437536 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:29.437561 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:29.437615 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:29.441263 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:29.442078 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:29.442087 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:29.442097 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:29.442301 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:29.442744 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:29.442750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:29.442753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:29.443435 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:29.443783 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:29.444288 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 01:07:29.444961 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":708,"thread_start_us":571,"threads_started":1}
Jun 19 01:07:29 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:29 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:29.448966 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:29.439733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:39757)
0619 01:07:29.439931 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:29.439952 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:29.440061 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:29.440367 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:29.440371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:29.440721 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:29.440857 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:29.440865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.442019 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.442025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:29.443098 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:29.443111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.444253 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.444261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:29.444454 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:29.445028 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:29.445051 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:29.448685 (+ 3634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 01:07:29.449141 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:29.439810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39757 (local address 127.0.97.62:40369)
0619 01:07:29.440011 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:29.440017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:29.440034 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:07:29.440162 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:29.440167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:29.440242 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:29.440344 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:29.441010 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.441881 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.443249 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.443940 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.444066 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:29.448917 (+ 4851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39757: BlockingRecv error: recv got EOF from 127.0.97.3:39757 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 01:07:29 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:29 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:29.457674 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:29.449820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:56203)
0619 01:07:29.450090 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:29.450108 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:29.450209 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:29.450675 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:29.450678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:29.450963 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:29.451107 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:29.451114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.452084 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.452087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:29.452998 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:29.453008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.453705 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.453709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:29.453839 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:29.454405 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:29.454423 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:29.457348 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159}
W20260619 01:07:29.457882 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:29.449905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56203 (local address 127.0.97.62:40369)
0619 01:07:29.450136 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:29.450140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:29.450155 (+ 15us) server_negotiation.cc:407] Connection header received
0619 01:07:29.450459 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:29.450466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:29.450560 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:29.450638 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:29.451219 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.451971 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.453115 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.453585 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.453680 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:29.457587 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56203: BlockingRecv error: recv got EOF from 127.0.97.3:56203 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260619 01:07:29.458097 9632 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 01:07:29 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:29 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:29.499926 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:29.489762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:49553)
0619 01:07:29.490344 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:29.490362 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:29.490474 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:29.490759 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:29.490762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:29.491208 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:29.491410 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:29.491419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.492386 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.492394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:29.494319 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:29.494354 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.495107 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.495113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:29.495397 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:29.496139 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:29.496161 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:29.499553 (+ 3392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":336,"threads_started":1}
W20260619 01:07:29.499979 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:29.489849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49553 (local address 127.0.97.62:40369)
0619 01:07:29.490038 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:29.490043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:29.490455 (+ 412us) server_negotiation.cc:407] Connection header received
0619 01:07:29.490571 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:29.490575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:29.490637 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:29.490720 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:29.491534 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.492229 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.494484 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.494896 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.494994 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:29.499764 (+ 4770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49553: BlockingRecv error: recv got EOF from 127.0.97.1:49553 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 01:07:29 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:29 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:29.989710 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:29.975688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:45783)
0619 01:07:29.976247 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:29.976270 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:29.976403 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:29.976717 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:29.976720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:29.977195 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:29.977469 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:29.977480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.978528 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.978533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:29.979520 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:29.979532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.980218 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.980223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:29.980347 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:29.984770 (+ 4423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:29.984797 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:29.989326 (+ 4529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":288,"threads_started":1}
W20260619 01:07:29.989965 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:29.975900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45783 (local address 127.0.97.62:40369)
0619 01:07:29.976146 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:29.976154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:29.976357 (+ 203us) server_negotiation.cc:407] Connection header received
0619 01:07:29.976508 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:29.976511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:29.976573 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:29.976655 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:29.977604 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.978400 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.979655 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:29.980054 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:29.980145 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:29.989641 (+ 9496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45783: BlockingRecv error: recv got EOF from 127.0.97.2:45783 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 01:07:30 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:30 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:30.466954 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:30.458625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:51971)
0619 01:07:30.459212 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:30.459229 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:30.459340 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:30.459633 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:30.459636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:30.460093 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:30.460354 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:30.460365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.461505 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.461509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:30.462469 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:30.462480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.463130 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.463134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:30.463260 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:30.463817 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:30.463833 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:30.466708 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":352,"threads_started":1}
W20260619 01:07:30.467207 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:30.458716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51971 (local address 127.0.97.62:40369)
0619 01:07:30.458880 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:30.458893 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:30.459301 (+ 408us) server_negotiation.cc:407] Connection header received
0619 01:07:30.459447 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:30.459450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:30.459511 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:30.459593 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:30.460483 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.461339 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.462591 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.462989 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.463079 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:30.466952 (+ 3873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51971: BlockingRecv error: recv got EOF from 127.0.97.3:51971 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 01:07:30 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:30 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:30.510226 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:30.501290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:57579)
0619 01:07:30.501862 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:30.501879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:30.502050 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:30.502438 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:30.502443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:30.502922 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:30.503111 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:30.503124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.504540 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.504544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:30.505790 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:30.505802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.506485 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.506490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:30.506607 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:30.507188 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:30.507207 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:30.509986 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":294,"threads_started":1}
W20260619 01:07:30.510442 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:30.501463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57579 (local address 127.0.97.62:40369)
0619 01:07:30.501699 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:30.501718 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:30.501973 (+ 255us) server_negotiation.cc:407] Connection header received
0619 01:07:30.502185 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:30.502191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:30.502274 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:30.502386 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:30.503313 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.504357 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.505913 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.506306 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.506412 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:30.510191 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57579: BlockingRecv error: recv got EOF from 127.0.97.1:57579 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 01:07:30 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:30 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:30.998724 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:30.990798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:44801)
0619 01:07:30.991268 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:30.991291 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:30.991414 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:30.991755 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:30.991758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:30.992157 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:30.992295 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:30.992302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.993343 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.993348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:30.994291 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:30.994303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.994977 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.994981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:30.995086 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:30.995693 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:30.995712 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:30.998465 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":260,"threads_started":1}
W20260619 01:07:30.998888 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:30.990862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44801 (local address 127.0.97.62:40369)
0619 01:07:30.991104 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:30.991110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:30.991367 (+ 257us) server_negotiation.cc:407] Connection header received
0619 01:07:30.991546 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:30.991550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:30.991630 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:30.991716 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:30.992417 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.993164 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.994421 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:30.994841 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:30.994954 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:30.998678 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44801: BlockingRecv error: recv got EOF from 127.0.97.2:44801 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 01:07:31 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:31 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:31.478005 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:31.468018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:58337)
0619 01:07:31.468496 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:31.468513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:31.468629 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:31.468933 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:31.468936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:31.469336 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:31.469506 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:31.469514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:31.470898 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:31.470903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:31.472079 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:31.472093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:31.473278 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:31.473285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:31.473533 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:31.474278 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:31.474307 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:31.477756 (+ 3449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":260,"threads_started":1}
W20260619 01:07:31.478248 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:31.468218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58337 (local address 127.0.97.62:40369)
0619 01:07:31.468460 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:31.468467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:31.468595 (+ 128us) server_negotiation.cc:407] Connection header received
0619 01:07:31.468729 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:31.468733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:31.468806 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:31.468891 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:31.469664 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:31.470741 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:31.472221 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:31.472906 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:31.473036 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:31.477960 (+ 4924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58337: BlockingRecv error: recv got EOF from 127.0.97.3:58337 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 19 01:07:31 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:31 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:31.519697 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:31.511210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:57751)
0619 01:07:31.511760 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:31.511783 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:31.511896 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:31.512223 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:31.512226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:31.512626 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:31.512793 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:31.512801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:31.513872 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:31.513876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:31.514795 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:31.514806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:31.515514 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:31.515518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:31.515638 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:31.516251 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:31.516269 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:31.519320 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":313,"threads_started":1}
W20260619 01:07:31.519840 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:31.511258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57751 (local address 127.0.97.62:40369)
0619 01:07:31.511447 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:31.511452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:31.511855 (+ 403us) server_negotiation.cc:407] Connection header received
0619 01:07:31.512040 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:31.512044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:31.512105 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:31.512186 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:31.512981 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:31.513732 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:31.514945 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:31.515387 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:31.515487 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:31.519526 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57751: BlockingRecv error: recv got EOF from 127.0.97.1:57751 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 01:07:32 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:32 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:32.008579 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:31.999876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:51475)
0619 01:07:32.000473 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:32.000495 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:32.000678 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:32.001037 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:32.001042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:32.001594 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:32.001767 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:32.001776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.002954 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.002960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:32.003865 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:32.003876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.004570 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.004576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:32.004782 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:32.005421 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:32.005452 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:32.008320 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":336,"threads_started":1}
W20260619 01:07:32.008842 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:32.000023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51475 (local address 127.0.97.62:40369)
0619 01:07:32.000281 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:32.000288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:32.000591 (+ 303us) server_negotiation.cc:407] Connection header received
0619 01:07:32.000831 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:32.000835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:32.000897 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:32.001004 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:32.001924 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.002824 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.004008 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.004415 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.004578 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:32.008526 (+ 3948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51475: BlockingRecv error: recv got EOF from 127.0.97.2:51475 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 19 01:07:32 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:32 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:32.488107 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:32.479397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:41437)
0619 01:07:32.479945 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:32.479973 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:32.480086 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:32.480354 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:32.480357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:32.480680 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:32.480855 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:32.480864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.481953 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.481958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:32.482880 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:32.482894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.483646 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.483651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:32.483787 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:32.484582 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:32.484602 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:32.487738 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":291,"threads_started":1}
W20260619 01:07:32.488317 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:32.479861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41437 (local address 127.0.97.62:40369)
0619 01:07:32.480037 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:32.480042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:32.480060 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:07:32.480173 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:32.480177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:32.480252 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:32.480338 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:32.481006 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.481814 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.483053 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.483479 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.483588 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:32.488036 (+ 4448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41437: BlockingRecv error: recv got EOF from 127.0.97.3:41437 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 01:07:32 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:32 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:32.529107 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:32.520812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:50447)
0619 01:07:32.521347 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:32.521376 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:32.521514 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:32.521828 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:32.521832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:32.522191 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:32.522341 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:32.522349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.523331 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.523335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:32.524304 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:32.524316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.525132 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.525140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:32.525377 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:32.526031 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:32.526050 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:32.528870 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":273,"threads_started":1}
W20260619 01:07:32.529458 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:32.520879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50447 (local address 127.0.97.62:40369)
0619 01:07:32.521122 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:32.521129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:32.521458 (+ 329us) server_negotiation.cc:407] Connection header received
0619 01:07:32.521622 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:32.521625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:32.521690 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:32.521777 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:32.522465 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.523200 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.524463 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:32.524938 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:32.525114 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:32.529107 (+ 3993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50447: BlockingRecv error: recv got EOF from 127.0.97.1:50447 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 01:07:33 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:33 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:33.018661 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:33.009679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:41943)
0619 01:07:33.010519 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:33.010545 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:33.010687 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:33.011050 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:33.011055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:33.011567 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:33.011774 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:33.011783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.012854 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.012860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:33.013943 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:33.013955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.014668 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.014673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:33.014865 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:33.015516 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:33.015534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:33.018262 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":295,"threads_started":1}
W20260619 01:07:33.018738 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:33.009743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41943 (local address 127.0.97.62:40369)
0619 01:07:33.010059 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:33.010065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:33.010643 (+ 578us) server_negotiation.cc:407] Connection header received
0619 01:07:33.010811 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:33.010818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:33.010894 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:33.011027 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:33.011936 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.012720 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.014089 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.014519 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.014656 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:33.018506 (+ 3850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41943: BlockingRecv error: recv got EOF from 127.0.97.2:41943 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
Jun 19 01:07:33 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:33 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:33.498201 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:33.489105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:57649)
0619 01:07:33.489677 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:33.489695 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:33.489858 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:33.490221 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:33.490226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:33.490605 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:33.490751 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:33.490759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.492272 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.492275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:33.493416 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:33.493429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.494133 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.494138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:33.494328 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:33.494974 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:33.494993 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:33.497846 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":321,"threads_started":1}
W20260619 01:07:33.498348 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:33.489311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57649 (local address 127.0.97.62:40369)
0619 01:07:33.489546 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:33.489554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:33.489778 (+ 224us) server_negotiation.cc:407] Connection header received
0619 01:07:33.489988 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:33.489993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:33.490081 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:33.490189 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:33.490919 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.491996 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.493565 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.493979 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.494121 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:33.498077 (+ 3956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57649: BlockingRecv error: recv got EOF from 127.0.97.3:57649 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 01:07:33 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:33 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:33.538311 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:33.530222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:50671)
0619 01:07:33.530714 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:33.530731 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:33.530844 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:33.531129 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:33.531132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:33.531558 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:33.531719 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:33.531728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.532692 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.532695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:33.533709 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:33.533721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.534350 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.534354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:33.534471 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:33.535085 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:33.535102 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:33.538069 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":241,"threads_started":1}
W20260619 01:07:33.538605 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:33.530245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50671 (local address 127.0.97.62:40369)
0619 01:07:33.530526 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:33.530532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:33.530810 (+ 278us) server_negotiation.cc:407] Connection header received
0619 01:07:33.530939 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:33.530942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:33.531004 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:33.531089 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:33.531851 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.532572 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.533836 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:33.534224 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:33.534323 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:33.538336 (+ 4013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50671: BlockingRecv error: recv got EOF from 127.0.97.1:50671 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 01:07:34 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:34 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:34.028281 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:34.019688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:44703)
0619 01:07:34.020225 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:34.020244 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:34.020400 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:34.020733 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:34.020737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:34.021189 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:34.021450 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:34.021462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.022672 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.022680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:34.023605 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:34.023616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.024353 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.024359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:34.024539 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:34.025117 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:34.025135 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:34.028023 (+ 2888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":256,"threads_started":1}
W20260619 01:07:34.028491 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:34.019904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44703 (local address 127.0.97.62:40369)
0619 01:07:34.020138 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:34.020144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:34.020335 (+ 191us) server_negotiation.cc:407] Connection header received
0619 01:07:34.020532 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:34.020542 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:34.020604 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:34.020710 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:34.021592 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.022539 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.023745 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.024192 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.024328 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:34.028237 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44703: BlockingRecv error: recv got EOF from 127.0.97.2:44703 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 01:07:34 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:34 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:34.508152 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:34.499295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:54755)
0619 01:07:34.499854 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:34.499870 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:34.500041 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:34.500424 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:34.500429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:34.500888 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:34.501044 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:34.501054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.502165 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.502171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:34.503100 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:34.503111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.503841 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.503847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:34.504037 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:34.504702 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:34.504720 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:34.507899 (+ 3179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":297,"threads_started":1}
W20260619 01:07:34.508354 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:34.499367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54755 (local address 127.0.97.62:40369)
0619 01:07:34.499602 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:34.499611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:34.499968 (+ 357us) server_negotiation.cc:407] Connection header received
0619 01:07:34.500179 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:34.500185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:34.500284 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:34.500394 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:34.501197 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.502007 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.503253 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.503679 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.503821 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:34.508097 (+ 4276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54755: BlockingRecv error: recv got EOF from 127.0.97.3:54755 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 01:07:34 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:34 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:34.547969 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:34.539257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:35481)
0619 01:07:34.539853 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:34.539872 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:34.540043 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:34.540449 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:34.540454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:34.540848 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:34.541059 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:34.541071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.542398 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.542403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:34.543560 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:34.543572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.544270 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.544275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:34.544406 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:34.544992 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:34.545010 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:34.547745 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":346,"threads_started":1}
W20260619 01:07:34.548269 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:34.539459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35481 (local address 127.0.97.62:40369)
0619 01:07:34.539713 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:34.539719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:34.539966 (+ 247us) server_negotiation.cc:407] Connection header received
0619 01:07:34.540199 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:34.540209 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:34.540269 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:34.540378 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:34.541216 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.542255 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.543686 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:34.544137 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:34.544235 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:34.547962 (+ 3727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35481: BlockingRecv error: recv got EOF from 127.0.97.1:35481 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 01:07:35 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:35 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:35.041777 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:35.029273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:56573)
0619 01:07:35.029857 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:35.029879 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:35.030014 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:35.030286 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:35.030289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:35.030664 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:35.030816 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:35.030824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.031774 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.031779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:35.032686 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:35.032697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.033409 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.033414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:35.033535 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:35.038430 (+ 4895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:35.038452 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:35.041539 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":322,"threads_started":1}
W20260619 01:07:35.042122 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:35.029467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56573 (local address 127.0.97.62:40369)
0619 01:07:35.029745 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:35.029751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:35.029965 (+ 214us) server_negotiation.cc:407] Connection header received
0619 01:07:35.030121 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:35.030125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:35.030188 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:35.030271 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:35.030961 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.031654 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.032820 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.033228 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.033399 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:35.041850 (+ 8451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56573: BlockingRecv error: recv got EOF from 127.0.97.2:56573 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 01:07:35 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:35 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:35.518544 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:35.509254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:40761)
0619 01:07:35.509899 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:35.509917 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:35.510081 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:35.510557 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:35.510561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:35.510992 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:35.511203 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:35.511216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.512416 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.512422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:35.513772 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:35.513786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.514517 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.514522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:35.514652 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:35.515277 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:35.515296 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:35.518278 (+ 2982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":314,"threads_started":1}
W20260619 01:07:35.518776 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:35.509369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40761 (local address 127.0.97.62:40369)
0619 01:07:35.509536 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:35.509542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:35.510009 (+ 467us) server_negotiation.cc:407] Connection header received
0619 01:07:35.510207 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:35.510211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:35.510341 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:35.510435 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:35.511361 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.512271 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.513910 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.514370 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.514469 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:35.518515 (+ 4046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40761: BlockingRecv error: recv got EOF from 127.0.97.3:40761 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 01:07:35 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:35 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:35.558331 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:35.549221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:38069)
0619 01:07:35.549907 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:35.549924 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:35.550096 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:35.550479 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:35.550489 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:35.551057 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:35.551300 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:35.551310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.552341 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.552345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:35.553449 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:35.553464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.554288 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.554293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:35.554416 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:35.555082 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:35.555101 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:35.558067 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":275,"threads_started":1}
W20260619 01:07:35.558506 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:35.549392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38069 (local address 127.0.97.62:40369)
0619 01:07:35.549593 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:35.549599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:35.550019 (+ 420us) server_negotiation.cc:407] Connection header received
0619 01:07:35.550228 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:35.550232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:35.550298 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:35.550409 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:35.551436 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.552188 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.553656 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:35.554138 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:35.554240 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:35.558300 (+ 4060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38069: BlockingRecv error: recv got EOF from 127.0.97.1:38069 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 01:07:36 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:36 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:36.051716 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:36.042814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:46587)
0619 01:07:36.043458 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:36.043482 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:36.043622 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:36.043982 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:36.043985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:36.044358 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:36.044527 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:36.044536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.045742 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.045747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:36.046694 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:36.046707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.047422 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.047427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:36.047543 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:36.048262 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:36.048280 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:36.051352 (+ 3072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":292,"threads_started":1}
W20260619 01:07:36.051853 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:36.042862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46587 (local address 127.0.97.62:40369)
0619 01:07:36.043119 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:36.043130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:36.043583 (+ 453us) server_negotiation.cc:407] Connection header received
0619 01:07:36.043770 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:36.043775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:36.043838 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:36.043927 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:36.044689 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.045576 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.046856 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.047259 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.047361 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:36.051575 (+ 4214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46587: BlockingRecv error: recv got EOF from 127.0.97.2:46587 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 19 01:07:36 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:36 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:36.528288 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:36.519705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:52651)
0619 01:07:36.520401 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:36.520428 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:36.520618 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:36.521005 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:36.521010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:36.521541 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:36.521721 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:36.521730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.522713 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.522716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:36.523650 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:36.523661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.524365 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.524369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:36.524508 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:36.525078 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:36.525095 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:36.528006 (+ 2911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":317,"threads_started":1}
W20260619 01:07:36.528508 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:36.519764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52651 (local address 127.0.97.62:40369)
0619 01:07:36.519946 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:36.519952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:36.520569 (+ 617us) server_negotiation.cc:407] Connection header received
0619 01:07:36.520754 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:36.520759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:36.520854 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:36.520958 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:36.521856 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.522588 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.523782 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.524229 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.524330 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:36.528251 (+ 3921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52651: BlockingRecv error: recv got EOF from 127.0.97.3:52651 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 01:07:36 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:36 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:36.567940 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:36.559326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:35263)
0619 01:07:36.559804 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:36.559824 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:36.559945 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:36.560286 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:36.560294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:36.560697 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:36.560892 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:36.560901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.561964 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.561968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:36.562917 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:36.562928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.563759 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.563764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:36.563886 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:36.564665 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:36.564683 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:36.567708 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":309,"threads_started":1}
W20260619 01:07:36.568295 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:36.559561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35263 (local address 127.0.97.62:40369)
0619 01:07:36.559754 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:36.559759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:36.559909 (+ 150us) server_negotiation.cc:407] Connection header received
0619 01:07:36.560046 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:36.560050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:36.560115 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:36.560215 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:36.561030 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.561807 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.563078 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:36.563568 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:36.563685 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:36.567954 (+ 4269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35263: BlockingRecv error: recv got EOF from 127.0.97.1:35263 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 01:07:37 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:37 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:37.062139 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:37.052848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:46851)
0619 01:07:37.053380 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:37.053409 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:37.053564 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:37.053991 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:37.053999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:37.054562 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:37.054759 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:37.054770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.056269 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.056275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:37.057283 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:37.057352 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.058111 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.058118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:37.058307 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:37.058866 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:37.058883 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:37.061801 (+ 2918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":687,"threads_started":1}
W20260619 01:07:37.062239 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:37.053042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46851 (local address 127.0.97.62:40369)
0619 01:07:37.053222 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:37.053226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:37.053492 (+ 266us) server_negotiation.cc:407] Connection header received
0619 01:07:37.053715 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:37.053723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:37.053821 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:37.053936 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:37.054917 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.056122 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.057493 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.057917 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.058052 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:37.062015 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46851: BlockingRecv error: recv got EOF from 127.0.97.2:46851 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 01:07:37 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:37 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:37.538199 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:37.529398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:42847)
0619 01:07:37.529890 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:37.529908 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:37.530027 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:37.530344 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:37.530347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:37.530734 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:37.530899 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:37.530907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.531996 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.531999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:37.532952 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:37.532964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.533740 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.533745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:37.533870 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:37.534505 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:37.534522 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:37.537964 (+ 3442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":292,"threads_started":1}
W20260619 01:07:37.538417 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:37.529465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42847 (local address 127.0.97.62:40369)
0619 01:07:37.529728 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:37.529735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:37.529995 (+ 260us) server_negotiation.cc:407] Connection header received
0619 01:07:37.530150 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:37.530154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:37.530216 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:37.530307 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:37.531107 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.531858 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.533095 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.533566 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.533677 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:37.538165 (+ 4488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42847: BlockingRecv error: recv got EOF from 127.0.97.3:42847 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 01:07:37 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:37 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:37.577732 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:37.568964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:59429)
0619 01:07:37.569572 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:37.569588 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:37.569735 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:37.570034 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:37.570036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:37.570409 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:37.570576 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:37.570584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.571543 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.571546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:37.572635 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:37.572647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.573685 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.573695 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:37.573856 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:37.574553 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:37.574571 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:37.577477 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":357,"threads_started":1}
W20260619 01:07:37.577921 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:37.569020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59429 (local address 127.0.97.62:40369)
0619 01:07:37.569198 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:37.569203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:37.569679 (+ 476us) server_negotiation.cc:407] Connection header received
0619 01:07:37.569838 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:37.569842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:37.569903 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:37.569993 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:37.570727 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.571428 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.572781 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:37.573505 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:37.573648 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:37.577693 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59429: BlockingRecv error: recv got EOF from 127.0.97.1:59429 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 01:07:38 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:38 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:38.072266 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:38.063282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:46525)
0619 01:07:38.063814 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:38.063835 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:38.063964 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:38.064279 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:38.064282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:38.064675 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:38.064838 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:38.064846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.065905 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.065909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:38.066926 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:38.066947 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.067770 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.067775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:38.067969 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:38.068744 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:38.068767 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:38.072023 (+ 3256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":294,"threads_started":1}
W20260619 01:07:38.072474 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:38.063518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46525 (local address 127.0.97.62:40369)
0619 01:07:38.063759 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:38.063768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:38.063947 (+ 179us) server_negotiation.cc:407] Connection header received
0619 01:07:38.064076 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:38.064080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:38.064148 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:38.064235 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:38.064974 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.065785 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.067120 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.067565 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.067708 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:38.072217 (+ 4509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46525: BlockingRecv error: recv got EOF from 127.0.97.2:46525 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 01:07:38 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:38 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:38.548877 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:38.539252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:39289)
0619 01:07:38.539834 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:38.539851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:38.539977 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:38.540340 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:38.540344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:38.540753 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:38.540928 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:38.540936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.542261 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.542266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:38.543297 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:38.543310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.544084 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.544090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:38.544225 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:38.544913 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:38.544934 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:38.548624 (+ 3690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":311,"threads_started":1}
W20260619 01:07:38.549243 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:38.539408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39289 (local address 127.0.97.62:40369)
0619 01:07:38.539657 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:38.539663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:38.539977 (+ 314us) server_negotiation.cc:407] Connection header received
0619 01:07:38.540071 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:38.540089 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:38.540168 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:38.540272 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:38.541105 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.542084 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.543457 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.543912 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.544020 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:38.548900 (+ 4880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39289: BlockingRecv error: recv got EOF from 127.0.97.3:39289 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 19 01:07:38 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:38 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:38.587724 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:38.578701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:60409)
0619 01:07:38.579183 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:38.579200 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:38.579317 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:38.579636 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:38.579639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:38.579987 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:38.580147 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:38.580156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.581210 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.581214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:38.582192 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:38.582204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.582959 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.582963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:38.583083 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:38.583866 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:38.583890 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:38.587491 (+ 3601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":256,"threads_started":1}
W20260619 01:07:38.588132 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:38.578906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60409 (local address 127.0.97.62:40369)
0619 01:07:38.579137 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:38.579146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:38.579279 (+ 133us) server_negotiation.cc:407] Connection header received
0619 01:07:38.579419 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:38.579423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:38.579483 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:38.579597 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:38.580291 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.581071 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.582338 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:38.582810 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:38.582929 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:38.587722 (+ 4793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60409: BlockingRecv error: recv got EOF from 127.0.97.1:60409 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 01:07:39 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:39 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:39.082569 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:39.073469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:45109)
0619 01:07:39.074081 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:39.074110 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:39.074237 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:39.074630 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:39.074634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:39.075175 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:39.075359 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:39.075370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.076680 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.076688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:39.077807 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:39.077820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.078529 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.078535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:39.078713 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:39.079371 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:39.079388 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:39.082238 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":354,"threads_started":1}
W20260619 01:07:39.082712 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:39.073607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45109 (local address 127.0.97.62:40369)
0619 01:07:39.073801 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:39.073806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:39.074213 (+ 407us) server_negotiation.cc:407] Connection header received
0619 01:07:39.074343 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:39.074347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:39.074410 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:39.074537 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:39.075562 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.076538 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.077959 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.078369 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.078511 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:39.082451 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45109: BlockingRecv error: recv got EOF from 127.0.97.2:45109 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 01:07:39 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:39 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:39.558770 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:39.550053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:55003)
0619 01:07:39.550599 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:39.550619 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:39.550792 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:39.551174 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:39.551178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:39.551678 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:39.551868 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:39.551882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.552884 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.552887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:39.553860 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:39.553871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.554561 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.554565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:39.554693 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:39.555369 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:39.555386 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:39.558407 (+ 3021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":410,"threads_started":1}
W20260619 01:07:39.558897 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:39.550224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55003 (local address 127.0.97.62:40369)
0619 01:07:39.550455 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:39.550460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:39.550723 (+ 263us) server_negotiation.cc:407] Connection header received
0619 01:07:39.550918 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:39.550922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:39.550999 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:39.551125 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:39.552029 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.552756 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.553999 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.554421 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.554520 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:39.558626 (+ 4106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55003: BlockingRecv error: recv got EOF from 127.0.97.3:55003 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Jun 19 01:07:39 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:39 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:39.597512 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:39.588761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:39467)
0619 01:07:39.589450 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:39.589466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:39.589619 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:39.589971 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:39.589976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:39.590435 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:39.590614 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:39.590623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.591635 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.591640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:39.592597 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:39.592610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.593422 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.593428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:39.593594 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:39.594242 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:39.594260 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:39.597187 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":286,"threads_started":1}
W20260619 01:07:39.597795 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:39.588881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39467 (local address 127.0.97.62:40369)
0619 01:07:39.589078 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:39.589086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:39.589545 (+ 459us) server_negotiation.cc:407] Connection header received
0619 01:07:39.589771 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:39.589775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:39.589838 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:39.589941 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:39.590757 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.591491 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.592758 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:39.593212 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:39.593373 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:39.597516 (+ 4143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39467: BlockingRecv error: recv got EOF from 127.0.97.1:39467 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 01:07:40 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:40 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:40.096895 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:40.083717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:39625)
0619 01:07:40.084230 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:40.084251 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:40.084377 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:40.084708 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:40.084711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:40.085118 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:40.085419 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:40.085431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.086431 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.086434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:40.087359 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:40.087370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.088103 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.088108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:40.088239 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:40.093712 (+ 5473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:40.093733 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:40.096630 (+ 2897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
W20260619 01:07:40.097112 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:40.083814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39625 (local address 127.0.97.62:40369)
0619 01:07:40.084056 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:40.084063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:40.084342 (+ 279us) server_negotiation.cc:407] Connection header received
0619 01:07:40.084484 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:40.084488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:40.084554 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:40.084642 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:40.085560 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.086305 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.087477 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.087902 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.088031 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:40.096883 (+ 8852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39625: BlockingRecv error: recv got EOF from 127.0.97.2:39625 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 01:07:40 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:40 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:40.570626 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:40.559982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:54493)
0619 01:07:40.560622 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:40.560642 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:40.560795 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:40.561168 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:40.561171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:40.561644 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:40.561824 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:40.561833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.563530 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.563534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:40.564492 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:40.564505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.565814 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.565822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:40.566019 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:40.566768 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:40.566793 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:40.570364 (+ 3571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":396,"threads_started":1}
W20260619 01:07:40.571064 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:40.560041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54493 (local address 127.0.97.62:40369)
0619 01:07:40.560230 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:40.560238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:40.560744 (+ 506us) server_negotiation.cc:407] Connection header received
0619 01:07:40.560934 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:40.560943 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:40.561038 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:40.561151 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:40.561969 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.563347 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.564687 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.565570 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.565715 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:40.570748 (+ 5033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54493: BlockingRecv error: recv got EOF from 127.0.97.3:54493 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 01:07:40 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:40 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 01:07:40.610349 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:40.598607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60173 (local address 127.0.97.62:40369)
0619 01:07:40.598856 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:40.598863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:40.599485 (+ 622us) server_negotiation.cc:407] Connection header received
0619 01:07:40.599650 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:40.599658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:40.599758 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:40.599882 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:40.600830 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.602347 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.603834 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.604765 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.604945 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:40.609912 (+ 4967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60173: BlockingRecv error: recv got EOF from 127.0.97.1:60173 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 01:07:40.610518 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:40.598597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:60173)
0619 01:07:40.599360 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:40.599380 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:40.599521 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:40.600021 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:40.600025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:40.600417 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:40.600649 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:40.600658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.602577 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.602582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:40.603644 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:40.603662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:40.604967 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:40.604972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:40.605154 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:40.606107 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:40.606127 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:40.610153 (+ 4026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":453,"threads_started":1}
Jun 19 01:07:41 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:41 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:41.107491 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:41.098061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:36615)
0619 01:07:41.098671 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:41.098699 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:41.098826 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:41.099144 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:41.099147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:41.099531 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:41.099716 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:41.099725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.100764 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.100769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:41.101789 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:41.101805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.102729 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.102736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:41.102925 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:41.103789 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:41.103811 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:41.107244 (+ 3433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":342,"threads_started":1}
W20260619 01:07:41.107677 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:41.098149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36615 (local address 127.0.97.62:40369)
0619 01:07:41.098371 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:41.098376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:41.098799 (+ 423us) server_negotiation.cc:407] Connection header received
0619 01:07:41.098933 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:41.098937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:41.099001 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:41.099095 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:41.099867 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.100581 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.101957 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.102496 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.102637 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:41.107464 (+ 4827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36615: BlockingRecv error: recv got EOF from 127.0.97.2:36615 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 01:07:41 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:41 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:41.580631 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:41.571934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:44241)
0619 01:07:41.572452 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:41.572469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:41.572660 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:41.573026 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:41.573031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:41.573455 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:41.573607 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:41.573615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.574654 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.574660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:41.575598 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:41.575610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.576370 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.576375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:41.576558 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:41.577170 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:41.577190 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:41.580284 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":279,"threads_started":1}
W20260619 01:07:41.580812 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:41.572009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44241 (local address 127.0.97.62:40369)
0619 01:07:41.572254 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:41.572262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:41.572589 (+ 327us) server_negotiation.cc:407] Connection header received
0619 01:07:41.572800 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:41.572805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:41.572874 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:41.572983 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:41.573756 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.574494 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.575746 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.576199 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.576340 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:41.580548 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44241: BlockingRecv error: recv got EOF from 127.0.97.3:44241 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 01:07:41 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:41 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:41.620488 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:41.611529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:43673)
0619 01:07:41.612120 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:41.612137 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:41.612273 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:41.612638 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:41.612642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:41.613047 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:41.613236 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:41.613252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.614361 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.614365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:41.615341 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:41.615354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.616116 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.616128 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:41.616276 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:41.616992 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:41.617018 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:41.620217 (+ 3199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":324,"threads_started":1}
W20260619 01:07:41.620832 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:41.611634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43673 (local address 127.0.97.62:40369)
0619 01:07:41.611839 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:41.611843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:41.612225 (+ 382us) server_negotiation.cc:407] Connection header received
0619 01:07:41.612403 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:41.612407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:41.612472 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:41.612625 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:41.613393 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.614203 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.615477 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:41.615924 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:41.616029 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:41.620438 (+ 4409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43673: BlockingRecv error: recv got EOF from 127.0.97.1:43673 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 01:07:42 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:42 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:42.117587 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:42.108608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:55715)
0619 01:07:42.109173 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:42.109194 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:42.109421 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:42.109796 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:42.109801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:42.110218 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:42.110416 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:42.110425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.111475 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.111482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:42.112469 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:42.112481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.113315 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.113322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:42.113520 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:42.114166 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:42.114184 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:42.117195 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":300,"threads_started":1}
W20260619 01:07:42.117663 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:42.108773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55715 (local address 127.0.97.62:40369)
0619 01:07:42.108939 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:42.108943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:42.109328 (+ 385us) server_negotiation.cc:407] Connection header received
0619 01:07:42.109563 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:42.109568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:42.109639 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:42.109772 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:42.110574 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.111327 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.112625 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.113078 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.113215 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:42.117420 (+ 4205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55715: BlockingRecv error: recv got EOF from 127.0.97.2:55715 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 01:07:42 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:42 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:42.591198 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:42.581808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:36333)
0619 01:07:42.582329 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:42.582348 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:42.582473 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:42.582859 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:42.582863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:42.583255 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:42.583452 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:42.583461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.584616 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.584620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:42.585872 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:42.585891 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.586919 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.586927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:42.587113 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:42.587708 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:42.587726 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:42.590949 (+ 3223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":272,"threads_started":1}
W20260619 01:07:42.591536 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:42.581854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36333 (local address 127.0.97.62:40369)
0619 01:07:42.582129 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:42.582136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:42.582444 (+ 308us) server_negotiation.cc:407] Connection header received
0619 01:07:42.582593 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:42.582597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:42.582672 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:42.582783 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:42.583629 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.584422 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.586033 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.586711 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.586850 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:42.591236 (+ 4386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36333: BlockingRecv error: recv got EOF from 127.0.97.3:36333 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 01:07:42 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:42 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:42.630270 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:42.621575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:34737)
0619 01:07:42.622106 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:42.622143 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:42.622314 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:42.622952 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:42.622957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:42.623391 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:42.623570 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:42.623603 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.624760 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.624765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:42.625785 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:42.625797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.626466 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.626470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:42.626601 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:42.627216 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:42.627234 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:42.630026 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":238,"threads_started":1}
W20260619 01:07:42.630509 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:42.621746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34737 (local address 127.0.97.62:40369)
0619 01:07:42.622061 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:42.622068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:42.622240 (+ 172us) server_negotiation.cc:407] Connection header received
0619 01:07:42.622673 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:42.622677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:42.622736 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:42.622866 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:42.623742 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.624615 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.625916 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:42.626330 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:42.626434 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:42.630270 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34737: BlockingRecv error: recv got EOF from 127.0.97.1:34737 (error 108)
Metrics: {"server-negotiator.queue_time_us":189}
Jun 19 01:07:43 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:43 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:43.127429 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:43.118627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:55383)
0619 01:07:43.119193 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:43.119229 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:43.119391 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:43.119746 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:43.119753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:43.120151 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:43.120303 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:43.120311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.121393 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.121399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:43.122346 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:43.122357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.123093 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.123100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:43.123286 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:43.123984 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:43.124002 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:43.127057 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":268,"threads_started":1}
W20260619 01:07:43.127552 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:43.118845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55383 (local address 127.0.97.62:40369)
0619 01:07:43.119065 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:43.119071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:43.119312 (+ 241us) server_negotiation.cc:407] Connection header received
0619 01:07:43.119519 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:43.119524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:43.119596 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:43.119710 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:43.120454 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.121201 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.122495 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.122928 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.123065 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:43.127324 (+ 4259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55383: BlockingRecv error: recv got EOF from 127.0.97.2:55383 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 01:07:43 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:43 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:43.600764 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:43.592335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.3:40271)
0619 01:07:43.592836 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:43.592859 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:43.593022 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:43.593474 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:43.593479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:43.593963 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:43.594116 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:43.594125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.595156 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.595160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:43.596090 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:43.596101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.596827 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.596831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:43.596946 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:43.597633 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:43.597650 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:43.600496 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":269,"threads_started":1}
W20260619 01:07:43.601006 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:43.592393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40271 (local address 127.0.97.62:40369)
0619 01:07:43.592593 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:43.592599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:43.592950 (+ 351us) server_negotiation.cc:407] Connection header received
0619 01:07:43.593168 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:43.593173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:43.593250 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:43.593388 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:43.594295 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.595022 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.596221 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.596638 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.596743 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:43.600767 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40271: BlockingRecv error: recv got EOF from 127.0.97.3:40271 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 01:07:43 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:43 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:43.640323 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:43.631506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.1:60283)
0619 01:07:43.632038 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:43.632054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:43.632178 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:43.632549 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:43.632554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:43.632999 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:43.633174 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:43.633183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.634354 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.634357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:43.635676 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:43.635690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.636446 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.636450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:43.636557 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:43.637155 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:43.637173 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:43.640055 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":297,"threads_started":1}
W20260619 01:07:43.640524 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:43.631552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60283 (local address 127.0.97.62:40369)
0619 01:07:43.631727 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:43.631732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:43.632127 (+ 395us) server_negotiation.cc:407] Connection header received
0619 01:07:43.632276 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:43.632291 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:43.632353 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:43.632439 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:43.633434 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.634198 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.635841 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:43.636304 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:43.636409 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:43.640300 (+ 3891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60283: BlockingRecv error: recv got EOF from 127.0.97.1:60283 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 01:07:44 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:07:44 dist-test-slave-5p8w krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:07:44.138264 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:44.128848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40369 (local address 127.0.97.2:33381)
0619 01:07:44.129522 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:44.129571 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:44.129719 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:44.130076 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:44.130079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:44.130481 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:44.130656 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:44.130664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:44.131807 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:44.131811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:44.132782 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:44.132795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:44.133603 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:44.133608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:44.133738 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:44.134530 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:44.134554 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:44.137980 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40369: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":332,"threads_started":1}
W20260619 01:07:44.138464 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:44.128989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33381 (local address 127.0.97.62:40369)
0619 01:07:44.129364 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:44.129370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:44.129661 (+ 291us) server_negotiation.cc:407] Connection header received
0619 01:07:44.129860 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:44.129865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:44.129931 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:44.130023 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:44.130788 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:44.131626 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:44.132932 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:44.133460 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:44.133566 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:44.138211 (+ 4645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33381: BlockingRecv error: recv got EOF from 127.0.97.2:33381 (error 108)
Metrics: {"server-negotiator.queue_time_us":185}
I20260619 01:07:44.438020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 9221
I20260619 01:07:44.451671 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:44.439123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.0.97.1:36839)
0619 01:07:44.439727 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:44.439735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:44.439763 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:07:44.439832 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:44.439839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:44.440067 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:44.440240 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:44.441755 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:44.442853 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:44.444099 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:44.444691 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:44.444806 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:44.448264 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:44.448279 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:44.448294 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:44.448360 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:44.451116 (+ 2756us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 01:07:44.451419 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57744: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":322,"threads_started":1}
W20260619 01:07:44.451793 9682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57744: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 01:07:44.451975 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 9221: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:36839: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 01:07:44.452190 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 9221
I20260619 01:07:44.470156 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 9373
I20260619 01:07:44.482219 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:44.471089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.0.97.2:36863)
0619 01:07:44.471691 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:44.471699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:44.471732 (+ 33us) server_negotiation.cc:407] Connection header received
0619 01:07:44.471796 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:44.471803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:44.472054 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:44.472219 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:44.473503 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:44.474615 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:44.475798 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:44.476367 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:44.476471 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:44.479011 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:44.479020 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:44.479035 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:44.479079 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:44.481680 (+ 2601us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 01:07:44.481964 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58964: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":293,"threads_started":1}
W20260619 01:07:44.482354 9683 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58964: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 01:07:44.482451 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 9373: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:36863: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 01:07:44.482630 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 9373
I20260619 01:07:44.499547 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 9504
I20260619 01:07:44.512223 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:44.500735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.0.97.3:44021)
0619 01:07:44.501387 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:44.501395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:44.501422 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:07:44.501491 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:44.501498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:44.501720 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:44.501884 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:44.503186 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:44.504210 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:44.505491 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:44.506053 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:44.506152 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:44.508748 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:44.508757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:44.508772 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:44.508814 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:44.511696 (+ 2882us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 01:07:44.511981 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43076: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":299,"threads_started":1}
W20260619 01:07:44.512411 9684 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43076: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 01:07:44.512519 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 9504: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:44021: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 01:07:44.512673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 9504
I20260619 01:07:44.530119 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 9152
I20260619 01:07:44.531525 9214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:44.647882 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 9152
2026-06-19T01:07:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21406 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T01:07:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:07:44Z Disabled control of system clock
I20260619 01:07:44.727933 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44485
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 01:07:45.072360 9689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:45.072696 9689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:45.072839 9689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:45.080479 9689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:07:45.080610 9689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:45.080681 9689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:45.080798 9689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:07:45.080873 9689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:07:45.090764 9689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:45.092250 9689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:45.094182 9689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:45.107678 9695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:45.107766 9694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:45.107717 9697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:45.113582 9689 server_base.cc:1034] running on GCE node
I20260619 01:07:45.114527 9689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:45.116255 9689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:45.117511 9689 hybrid_clock.cc:648] HybridClock initialized: now 1781831265117461 us; error 49 us; skew 500 ppm
I20260619 01:07:45.117910 9689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:07:45.127205 9689 webserver.cc:460] Webserver started at http://127.0.97.62:46669/ using document root <none> and password file <none>
I20260619 01:07:45.128057 9689 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:45.128193 9689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:45.128568 9689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:45.133555 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "d0152e4c37a24fc9befac1185442254b"
format_stamp: "Formatted at 2026-06-19 01:07:45 on dist-test-slave-5p8w"
server_key: "5995302971145fd0b7b746098ffe5f00"
server_key_iv: "e23f6f6b532c5dff8e90259ea2fc2282"
server_key_version: "encryptionkey@0"
I20260619 01:07:45.134696 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "d0152e4c37a24fc9befac1185442254b"
format_stamp: "Formatted at 2026-06-19 01:07:45 on dist-test-slave-5p8w"
server_key: "5995302971145fd0b7b746098ffe5f00"
server_key_iv: "e23f6f6b532c5dff8e90259ea2fc2282"
server_key_version: "encryptionkey@0"
I20260619 01:07:45.141716 9689 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260619 01:07:45.145465 9703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:45.147001 9689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:45.147198 9689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "d0152e4c37a24fc9befac1185442254b"
format_stamp: "Formatted at 2026-06-19 01:07:45 on dist-test-slave-5p8w"
server_key: "5995302971145fd0b7b746098ffe5f00"
server_key_iv: "e23f6f6b532c5dff8e90259ea2fc2282"
server_key_version: "encryptionkey@0"
I20260619 01:07:45.147447 9689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:45.165069 9689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:45.166034 9689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:45.166352 9689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:45.186281 9689 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44485
I20260619 01:07:45.186303 9754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44485 every 8 connection(s)
I20260619 01:07:45.187772 9689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:07:45.191771 9755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:45.192353 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 9689
I20260619 01:07:45.192579 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:07:45.193454 388 external_mini_cluster.cc:1428] Setting key 73bf1a035b3e75fa9d9d6c23a5d4752a
I20260619 01:07:45.205674 9755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b: Bootstrap starting.
I20260619 01:07:45.208220 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:45.195756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.0.97.62:44485)
0619 01:07:45.196827 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:45.196843 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:45.196888 (+ 45us) server_negotiation.cc:407] Connection header received
0619 01:07:45.198749 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:45.198775 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:45.199330 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:45.199753 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:45.200402 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:45.201752 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:45.203214 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:45.203994 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:45.204226 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:45.205629 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:45.205637 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:45.205655 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:45.206301 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:45.206401 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:45.206431 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:45.206568 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:45.206814 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 01:07:45.207168 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":304,"threads_started":1}
I20260619 01:07:45.209407 9755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:45.212034 9755 log.cc:824] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:45.215432 9755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b: No bootstrap required, opened a new log
I20260619 01:07:45.223650 9755 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0152e4c37a24fc9befac1185442254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44485 } }
I20260619 01:07:45.224086 9755 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:45.224161 9755 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0152e4c37a24fc9befac1185442254b, State: Initialized, Role: FOLLOWER
I20260619 01:07:45.224776 9755 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0152e4c37a24fc9befac1185442254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44485 } }
I20260619 01:07:45.225064 9755 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:07:45.225271 9755 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:07:45.225589 9755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:45.227916 9755 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0152e4c37a24fc9befac1185442254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44485 } }
I20260619 01:07:45.228502 9755 leader_election.cc:302] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0152e4c37a24fc9befac1185442254b; no voters:
I20260619 01:07:45.229171 9755 leader_election.cc:288] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:07:45.229337 9760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:45.230069 9760 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [term 1 LEADER]: Becoming Leader. State: Replica: d0152e4c37a24fc9befac1185442254b, State: Running, Role: LEADER
I20260619 01:07:45.230638 9760 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0152e4c37a24fc9befac1185442254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44485 } }
I20260619 01:07:45.231455 9755 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:07:45.233525 9762 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0152e4c37a24fc9befac1185442254b. Latest consensus state: current_term: 1 leader_uuid: "d0152e4c37a24fc9befac1185442254b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0152e4c37a24fc9befac1185442254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44485 } } }
I20260619 01:07:45.233562 9761 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0152e4c37a24fc9befac1185442254b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0152e4c37a24fc9befac1185442254b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44485 } } }
I20260619 01:07:45.233978 9762 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:45.233976 9761 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:45.237638 9765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:07:45.244192 9765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:07:45.255336 9765 catalog_manager.cc:1347] Generated new cluster ID: 4e4c3eb54bb846cca71df7999119faf4
I20260619 01:07:45.255483 9765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:07:45.267189 9765 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:07:45.268541 9765 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:07:45.275791 9765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0152e4c37a24fc9befac1185442254b: Generated new TSK 0
I20260619 01:07:45.276813 9765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:07:45.296204 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44485
--builtin_ntp_servers=127.0.97.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 01:07:45.671783 9779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:45.672107 9779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:45.672199 9779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:45.679586 9779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:45.679737 9779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:45.679852 9779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:07:45.689253 9779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:45.690559 9779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:45.692301 9779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:45.706104 9787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:45.706219 9785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:45.706197 9784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:45.707410 9779 server_base.cc:1034] running on GCE node
I20260619 01:07:45.708149 9779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:45.709220 9779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:45.710479 9779 hybrid_clock.cc:648] HybridClock initialized: now 1781831265710399 us; error 74 us; skew 500 ppm
I20260619 01:07:45.710824 9779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:07:45.713490 9779 webserver.cc:460] Webserver started at http://127.0.97.1:43353/ using document root <none> and password file <none>
I20260619 01:07:45.714283 9779 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:45.714435 9779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:45.714781 9779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:45.717998 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "3345b07642b246959812dfcdd88bfd94"
format_stamp: "Formatted at 2026-06-19 01:07:45 on dist-test-slave-5p8w"
server_key: "132b797a27a3413c0c3b5b92756cc7ee"
server_key_iv: "356c5b22b1d64a4c1e6adee7223c6545"
server_key_version: "encryptionkey@0"
I20260619 01:07:45.718854 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3345b07642b246959812dfcdd88bfd94"
format_stamp: "Formatted at 2026-06-19 01:07:45 on dist-test-slave-5p8w"
server_key: "132b797a27a3413c0c3b5b92756cc7ee"
server_key_iv: "356c5b22b1d64a4c1e6adee7223c6545"
server_key_version: "encryptionkey@0"
I20260619 01:07:45.724474 9779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260619 01:07:45.728312 9793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:45.729931 9779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:07:45.730135 9779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "3345b07642b246959812dfcdd88bfd94"
format_stamp: "Formatted at 2026-06-19 01:07:45 on dist-test-slave-5p8w"
server_key: "132b797a27a3413c0c3b5b92756cc7ee"
server_key_iv: "356c5b22b1d64a4c1e6adee7223c6545"
server_key_version: "encryptionkey@0"
I20260619 01:07:45.730384 9779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:45.744697 9779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:45.745709 9779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:45.746016 9779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:45.747015 9779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:45.748555 9779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:45.748682 9779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:45.748824 9779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:45.748921 9779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:45.783557 9779 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44753
I20260619 01:07:45.783600 9905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44753 every 8 connection(s)
I20260619 01:07:45.785005 9779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:07:45.795172 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 9779
I20260619 01:07:45.795439 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:07:45.796397 388 external_mini_cluster.cc:1428] Setting key 390153500d896b16261171b85f46edc4
I20260619 01:07:45.799608 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:45.787844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48555 (local address 127.0.97.62:44485)
0619 01:07:45.788330 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:45.788356 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:45.789828 (+ 1472us) server_negotiation.cc:407] Connection header received
0619 01:07:45.791576 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:45.791580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:45.791662 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:45.791763 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:45.792919 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:45.793745 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:45.796341 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:45.796814 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:45.796924 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:45.798604 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:45.798611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:45.798615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:45.798636 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:45.798687 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:45.798704 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:45.798813 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:45.799095 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 01:07:45.799199 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":218,"threads_started":1}
I20260619 01:07:45.799768 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44485
--builtin_ntp_servers=127.0.97.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 01:07:45.800657 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:45.788562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44485 (local address 127.0.97.1:48555)
0619 01:07:45.789628 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:45.789669 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:45.791312 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:45.792016 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:45.792024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:45.792122 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:45.792754 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:45.792772 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:45.793909 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:45.793913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:45.796112 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:45.796134 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:45.797084 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:45.797091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:45.797390 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:45.798175 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:07:45.798179 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:45.798251 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:07:45.798364 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:45.798830 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:45.798858 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:45.798865 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 01:07:45.799048 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 01:07:45.799400 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":281,"threads_started":1}
I20260619 01:07:45.802423 9906 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44485
I20260619 01:07:45.802784 9906 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:45.803800 9906 heartbeater.cc:511] Master 127.0.97.62:44485 requested a full tablet report, sending...
I20260619 01:07:45.806209 9720 ts_manager.cc:194] Registered new tserver with Master: 3345b07642b246959812dfcdd88bfd94 (127.0.97.1:44753)
I20260619 01:07:45.807734 9720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:48555
W20260619 01:07:46.147778 9910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:46.148094 9910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:46.148254 9910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:46.156081 9910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:46.156208 9910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:46.156391 9910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:07:46.165992 9910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:46.167438 9910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:46.169378 9910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:46.183629 9915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:46.183617 9916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:46.183837 9918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:46.184412 9910 server_base.cc:1034] running on GCE node
I20260619 01:07:46.185153 9910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:46.186543 9910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:46.187815 9910 hybrid_clock.cc:648] HybridClock initialized: now 1781831266187745 us; error 63 us; skew 500 ppm
I20260619 01:07:46.188233 9910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:07:46.191326 9910 webserver.cc:460] Webserver started at http://127.0.97.2:39947/ using document root <none> and password file <none>
I20260619 01:07:46.192353 9910 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:46.192480 9910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:46.192834 9910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:46.197026 9910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "7b88a17664b84ec9a62e822e4020eec3"
format_stamp: "Formatted at 2026-06-19 01:07:46 on dist-test-slave-5p8w"
server_key: "21d1679414e339328cd90990295fabfd"
server_key_iv: "47cbcb3dc7ce5ce9260e736dea0acd58"
server_key_version: "encryptionkey@0"
I20260619 01:07:46.198179 9910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7b88a17664b84ec9a62e822e4020eec3"
format_stamp: "Formatted at 2026-06-19 01:07:46 on dist-test-slave-5p8w"
server_key: "21d1679414e339328cd90990295fabfd"
server_key_iv: "47cbcb3dc7ce5ce9260e736dea0acd58"
server_key_version: "encryptionkey@0"
I20260619 01:07:46.205560 9910 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260619 01:07:46.210008 9924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:46.211553 9910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:46.211760 9910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "7b88a17664b84ec9a62e822e4020eec3"
format_stamp: "Formatted at 2026-06-19 01:07:46 on dist-test-slave-5p8w"
server_key: "21d1679414e339328cd90990295fabfd"
server_key_iv: "47cbcb3dc7ce5ce9260e736dea0acd58"
server_key_version: "encryptionkey@0"
I20260619 01:07:46.212021 9910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:46.229074 9910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:46.230058 9910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:46.230370 9910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:46.231408 9910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:46.233116 9910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:46.233245 9910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:46.233417 9910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:46.233484 9910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:46.268782 9910 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44335
I20260619 01:07:46.268833 10036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44335 every 8 connection(s)
I20260619 01:07:46.270249 9910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:07:46.276444 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 9910
I20260619 01:07:46.276641 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:07:46.277233 388 external_mini_cluster.cc:1428] Setting key 0bfb4dbe3ec91318a6f323ba037581d7
I20260619 01:07:46.280704 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44485
--builtin_ntp_servers=127.0.97.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 01:07:46.285745 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:46.273006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60437 (local address 127.0.97.62:44485)
0619 01:07:46.273193 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:46.273198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:46.274847 (+ 1649us) server_negotiation.cc:407] Connection header received
0619 01:07:46.277036 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:46.277041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:46.277114 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:46.277208 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:46.278600 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.279654 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.282525 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.282949 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.283061 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:46.284895 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:46.284901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:46.284904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:46.284925 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:46.284960 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:46.284972 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:46.285040 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:46.285413 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 01:07:46.285504 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 01:07:46.287433 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:46.273700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44485 (local address 127.0.97.2:60437)
0619 01:07:46.274650 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:46.274694 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:46.276691 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:46.277573 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:46.277587 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:46.277729 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:46.278432 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:46.278450 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.279847 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.279853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:46.282298 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:46.282323 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.283221 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.283231 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:46.283597 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:46.284439 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:07:46.284444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:46.284508 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:07:46.284634 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:46.285074 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:46.285088 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:46.285101 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 01:07:46.285339 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 01:07:46.285840 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":283,"threads_started":1}
I20260619 01:07:46.289135 10037 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44485
I20260619 01:07:46.289589 10037 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:46.290693 10037 heartbeater.cc:511] Master 127.0.97.62:44485 requested a full tablet report, sending...
I20260619 01:07:46.292484 9720 ts_manager.cc:194] Registered new tserver with Master: 7b88a17664b84ec9a62e822e4020eec3 (127.0.97.2:44335)
I20260619 01:07:46.293448 9720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:60437
W20260619 01:07:46.635409 10040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:46.635820 10040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:46.635984 10040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:46.643607 10040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:46.643721 10040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:46.643893 10040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:07:46.653710 10040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:46.655068 10040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:46.656915 10040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:46.670738 10045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:46.670696 10048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:46.670797 10046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:46.671816 10040 server_base.cc:1034] running on GCE node
I20260619 01:07:46.672504 10040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:46.673599 10040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:46.674839 10040 hybrid_clock.cc:648] HybridClock initialized: now 1781831266674783 us; error 55 us; skew 500 ppm
I20260619 01:07:46.675187 10040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:07:46.677726 10040 webserver.cc:460] Webserver started at http://127.0.97.3:36121/ using document root <none> and password file <none>
I20260619 01:07:46.678488 10040 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:46.678624 10040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:46.678967 10040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:46.681996 10040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "3a1e758e50ee4c5197c09f86db868999"
format_stamp: "Formatted at 2026-06-19 01:07:46 on dist-test-slave-5p8w"
server_key: "237fd12f9779a1ba65a6f803a787f62e"
server_key_iv: "4185b9567df18ec5329163b3a78bb463"
server_key_version: "encryptionkey@0"
I20260619 01:07:46.682826 10040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3a1e758e50ee4c5197c09f86db868999"
format_stamp: "Formatted at 2026-06-19 01:07:46 on dist-test-slave-5p8w"
server_key: "237fd12f9779a1ba65a6f803a787f62e"
server_key_iv: "4185b9567df18ec5329163b3a78bb463"
server_key_version: "encryptionkey@0"
I20260619 01:07:46.688202 10040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:07:46.691977 10054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:46.693523 10040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 01:07:46.693713 10040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "3a1e758e50ee4c5197c09f86db868999"
format_stamp: "Formatted at 2026-06-19 01:07:46 on dist-test-slave-5p8w"
server_key: "237fd12f9779a1ba65a6f803a787f62e"
server_key_iv: "4185b9567df18ec5329163b3a78bb463"
server_key_version: "encryptionkey@0"
I20260619 01:07:46.693965 10040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:46.714877 10040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:46.715831 10040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:46.716137 10040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:46.717188 10040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:46.718842 10040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:46.718971 10040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:46.719094 10040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:46.719179 10040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:46.753271 10040 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45429
I20260619 01:07:46.753472 10166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45429 every 8 connection(s)
I20260619 01:07:46.754817 10040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:07:46.759047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 10040
I20260619 01:07:46.759239 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:07:46.759797 388 external_mini_cluster.cc:1428] Setting key 0955fb05bd538b904f8cd2298daddc04
I20260619 01:07:46.768306 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:46.757483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48671 (local address 127.0.97.62:44485)
0619 01:07:46.757708 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:46.757715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:46.759568 (+ 1853us) server_negotiation.cc:407] Connection header received
0619 01:07:46.761758 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:46.761767 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:46.761863 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:46.761946 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:46.763287 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.764022 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.765734 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.766130 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.766234 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:46.767575 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:46.767580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:46.767583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:46.767606 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:46.767638 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:46.767650 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:46.767756 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:46.767970 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 01:07:46.768055 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:07:46.769474 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:46.758140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44485 (local address 127.0.97.3:48671)
0619 01:07:46.759367 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:46.759427 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:46.761384 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:46.762220 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:46.762229 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:46.762335 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:46.763123 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:46.763154 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.764157 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.764161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:46.765585 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:46.765598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.766337 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.766342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:46.766546 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:46.767245 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:07:46.767247 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:46.767291 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:07:46.767380 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:46.767756 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:46.767764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:46.767769 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 01:07:46.767947 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 01:07:46.768279 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":279,"threads_started":1}
I20260619 01:07:46.770999 10167 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44485
I20260619 01:07:46.771365 10167 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:46.772258 10167 heartbeater.cc:511] Master 127.0.97.62:44485 requested a full tablet report, sending...
I20260619 01:07:46.773860 9720 ts_manager.cc:194] Registered new tserver with Master: 3a1e758e50ee4c5197c09f86db868999 (127.0.97.3:45429)
I20260619 01:07:46.774838 9720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:48671
I20260619 01:07:46.775627 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 01:07:46.781905 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:46.780398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.0.97.62:44485)
0619 01:07:46.780674 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:46.780681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:46.780867 (+ 186us) server_negotiation.cc:407] Connection header received
0619 01:07:46.781014 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:46.781018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:46.781092 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:46.781177 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:46.781681 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54120: BlockingRecv error: recv got EOF from 127.0.0.1:54120 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 01:07:46.782541 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 9779
I20260619 01:07:46.791873 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:46.783617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47086 (local address 127.0.97.1:44753)
0619 01:07:46.784183 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:46.784191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:46.784218 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:07:46.784282 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:46.784288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:46.784507 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:46.784638 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:46.785073 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.786286 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.788163 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.788983 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.789090 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:46.790435 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:46.790445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:46.790458 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:46.791145 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:46.791193 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:46.791214 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:46.791319 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:46.791455 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 01:07:46.791576 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":279,"threads_started":1}
I20260619 01:07:46.793157 9901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:46.874595 9906 heartbeater.cc:503] Master 127.0.97.62:44485 was elected leader, sending a full tablet report...
I20260619 01:07:46.875910 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 9779
W20260619 01:07:46.892483 9707 connection.cc:535] server connection from 127.0.97.1:48555 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 01:07:46.892796 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 9910
I20260619 01:07:46.901262 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:46.893719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.0.97.2:44335)
0619 01:07:46.894312 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:46.894320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:46.894349 (+ 29us) server_negotiation.cc:407] Connection header received
0619 01:07:46.894417 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:46.894424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:46.894653 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:46.894878 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:46.895295 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.896297 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.897796 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:46.898411 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:46.898560 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:46.899725 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:46.899733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:46.899748 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:46.900386 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:46.900443 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:46.900462 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:46.900595 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:46.900755 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 01:07:46.900924 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":323,"threads_started":1}
I20260619 01:07:46.902649 10032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:46.984421 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 9910
I20260619 01:07:47.001082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 10040
I20260619 01:07:47.009394 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:47.002231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.0.97.3:45429)
0619 01:07:47.002887 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:47.002895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:47.002928 (+ 33us) server_negotiation.cc:407] Connection header received
0619 01:07:47.002992 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:47.002998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:47.003224 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:47.003379 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:47.003753 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:47.004797 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:47.006167 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:47.006797 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:47.006906 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:47.007926 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:47.007936 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:47.007950 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:47.008588 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:47.008644 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:47.008663 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:47.008760 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:47.008887 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 01:07:47.009017 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":348,"threads_started":1}
I20260619 01:07:47.010715 10162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:47.098106 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 10040
I20260619 01:07:47.114660 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 9689
I20260619 01:07:47.115916 9750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:47.221730 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 9689
2026-06-19T01:07:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2567 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T01:07:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:07:47Z Disabled control of system clock
I20260619 01:07:47.296078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44331
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 01:07:47.642091 10186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:47.642411 10186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:47.642566 10186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:47.650329 10186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:07:47.650453 10186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:47.650535 10186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:47.650640 10186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:07:47.650728 10186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:07:47.660720 10186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:47.662140 10186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:47.664002 10186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:47.676678 10191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:47.676878 10194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:47.676704 10192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:47.677740 10186 server_base.cc:1034] running on GCE node
I20260619 01:07:47.678597 10186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:47.680164 10186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:47.681414 10186 hybrid_clock.cc:648] HybridClock initialized: now 1781831267681347 us; error 55 us; skew 500 ppm
I20260619 01:07:47.681743 10186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:07:47.684132 10186 webserver.cc:460] Webserver started at http://127.0.97.62:40669/ using document root <none> and password file <none>
I20260619 01:07:47.684854 10186 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:47.684947 10186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:47.685220 10186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:47.689813 10186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "4d88d42ed1684839862da871f0de18c8"
format_stamp: "Formatted at 2026-06-19 01:07:47 on dist-test-slave-5p8w"
server_key: "f3dab8c143008f7928b4164ac0d7e435"
server_key_iv: "1a7a25215eb0fb15676e4543df540fb4"
server_key_version: "encryptionkey@0"
I20260619 01:07:47.690613 10186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "4d88d42ed1684839862da871f0de18c8"
format_stamp: "Formatted at 2026-06-19 01:07:47 on dist-test-slave-5p8w"
server_key: "f3dab8c143008f7928b4164ac0d7e435"
server_key_iv: "1a7a25215eb0fb15676e4543df540fb4"
server_key_version: "encryptionkey@0"
I20260619 01:07:47.696096 10186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:07:47.699772 10200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:47.701377 10186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:07:47.701548 10186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "4d88d42ed1684839862da871f0de18c8"
format_stamp: "Formatted at 2026-06-19 01:07:47 on dist-test-slave-5p8w"
server_key: "f3dab8c143008f7928b4164ac0d7e435"
server_key_iv: "1a7a25215eb0fb15676e4543df540fb4"
server_key_version: "encryptionkey@0"
I20260619 01:07:47.701745 10186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:47.707782 10186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:47.708647 10186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:47.708913 10186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:47.728173 10186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44331
I20260619 01:07:47.728194 10251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44331 every 8 connection(s)
I20260619 01:07:47.729753 10186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:07:47.730522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 10186
I20260619 01:07:47.730752 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:07:47.731424 388 external_mini_cluster.cc:1428] Setting key d9f092eb692aa553029e3c60eafdce1f
I20260619 01:07:47.734840 10252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:47.742007 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:47.733630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.0.97.62:44331)
0619 01:07:47.735359 (+ 1729us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:47.735375 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:47.735425 (+ 50us) server_negotiation.cc:407] Connection header received
0619 01:07:47.737639 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:47.737668 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:47.738270 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:47.738725 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:47.739787 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:47.739795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:47.739813 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:47.739831 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:47.739953 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:47.739976 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:47.740228 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:47.740361 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 01:07:47.740650 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172,"thread_start_us":933,"threads_started":1}
I20260619 01:07:47.748657 10252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8: Bootstrap starting.
I20260619 01:07:47.752313 10252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:47.753585 10252 log.cc:824] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:47.756997 10252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8: No bootstrap required, opened a new log
I20260619 01:07:47.764992 10252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d88d42ed1684839862da871f0de18c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44331 } }
I20260619 01:07:47.765491 10252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:47.765573 10252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d88d42ed1684839862da871f0de18c8, State: Initialized, Role: FOLLOWER
I20260619 01:07:47.766243 10252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d88d42ed1684839862da871f0de18c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44331 } }
I20260619 01:07:47.766450 10252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:07:47.766543 10252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:07:47.766673 10252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:47.769248 10252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d88d42ed1684839862da871f0de18c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44331 } }
I20260619 01:07:47.769819 10252 leader_election.cc:302] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d88d42ed1684839862da871f0de18c8; no voters:
I20260619 01:07:47.770442 10252 leader_election.cc:288] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:07:47.770619 10257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:47.771390 10257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [term 1 LEADER]: Becoming Leader. State: Replica: 4d88d42ed1684839862da871f0de18c8, State: Running, Role: LEADER
I20260619 01:07:47.771962 10257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d88d42ed1684839862da871f0de18c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44331 } }
I20260619 01:07:47.772887 10252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:07:47.775198 10259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d88d42ed1684839862da871f0de18c8. Latest consensus state: current_term: 1 leader_uuid: "4d88d42ed1684839862da871f0de18c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d88d42ed1684839862da871f0de18c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44331 } } }
I20260619 01:07:47.775317 10258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d88d42ed1684839862da871f0de18c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d88d42ed1684839862da871f0de18c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44331 } } }
I20260619 01:07:47.775601 10258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:47.775604 10259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:47.776507 10265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:07:47.784390 10265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:07:47.794395 10265 catalog_manager.cc:1347] Generated new cluster ID: cdbd65ce0b9a42b19fe913d0551decac
I20260619 01:07:47.794526 10265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:07:47.813995 10265 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:07:47.814478 10265 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:07:47.823958 10265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d88d42ed1684839862da871f0de18c8: Generated new TSK 0
I20260619 01:07:47.824947 10265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:07:47.832261 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44331
--builtin_ntp_servers=127.0.97.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 01:07:48.220165 10276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:48.220476 10276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:48.220635 10276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:48.228502 10276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:48.228626 10276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:48.228837 10276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:07:48.238317 10276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:48.239759 10276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:48.241632 10276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:48.255250 10281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:48.255307 10282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:48.255250 10284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:48.256852 10276 server_base.cc:1034] running on GCE node
I20260619 01:07:48.257735 10276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:48.258903 10276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:48.260149 10276 hybrid_clock.cc:648] HybridClock initialized: now 1781831268260102 us; error 53 us; skew 500 ppm
I20260619 01:07:48.260510 10276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:07:48.263072 10276 webserver.cc:460] Webserver started at http://127.0.97.1:41985/ using document root <none> and password file <none>
I20260619 01:07:48.263854 10276 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:48.263988 10276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:48.264312 10276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:48.267395 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "45e8bdc7bbd64143b89ea257674b63b7"
format_stamp: "Formatted at 2026-06-19 01:07:48 on dist-test-slave-5p8w"
server_key: "a42cf5086208ecc8772b6c0d2ffaa92b"
server_key_iv: "be1858662a5fceb7f0f92ceee432cc68"
server_key_version: "encryptionkey@0"
I20260619 01:07:48.268213 10276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "45e8bdc7bbd64143b89ea257674b63b7"
format_stamp: "Formatted at 2026-06-19 01:07:48 on dist-test-slave-5p8w"
server_key: "a42cf5086208ecc8772b6c0d2ffaa92b"
server_key_iv: "be1858662a5fceb7f0f92ceee432cc68"
server_key_version: "encryptionkey@0"
I20260619 01:07:48.274116 10276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 01:07:48.278039 10290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:48.279592 10276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:48.279834 10276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "45e8bdc7bbd64143b89ea257674b63b7"
format_stamp: "Formatted at 2026-06-19 01:07:48 on dist-test-slave-5p8w"
server_key: "a42cf5086208ecc8772b6c0d2ffaa92b"
server_key_iv: "be1858662a5fceb7f0f92ceee432cc68"
server_key_version: "encryptionkey@0"
I20260619 01:07:48.280093 10276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:48.286108 10276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:48.286949 10276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:48.287238 10276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:48.288174 10276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:48.289759 10276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:48.289878 10276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:48.290015 10276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:48.290102 10276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:48.324774 10276 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35115
I20260619 01:07:48.324819 10402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35115 every 8 connection(s)
I20260619 01:07:48.326207 10276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:07:48.331265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 10276
I20260619 01:07:48.331483 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:07:48.332283 388 external_mini_cluster.cc:1428] Setting key 8e06df224822c6e25d01462705d08301
I20260619 01:07:48.335814 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44331
--builtin_ntp_servers=127.0.97.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 01:07:48.336170 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:48.328909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34739 (local address 127.0.97.62:44331)
0619 01:07:48.329397 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:48.329404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:48.330795 (+ 1391us) server_negotiation.cc:407] Connection header received
0619 01:07:48.333451 (+ 2656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:48.333456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:48.333512 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:48.333652 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:48.335243 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:48.335251 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:48.335255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:48.335272 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:48.335316 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:48.335331 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:48.335437 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:48.335736 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 01:07:48.335815 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":218,"threads_started":1}
I20260619 01:07:48.337584 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:48.329605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44331 (local address 127.0.97.1:34739)
0619 01:07:48.330554 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:48.330602 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:48.333071 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:48.333885 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:48.333894 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:48.334008 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:48.334825 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:07:48.334828 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:48.334888 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:07:48.335034 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:48.335454 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:48.335467 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:48.335478 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 01:07:48.335804 (+ 326us) client_negotiation.cc:240] Negotiation successful
0619 01:07:48.336125 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":280,"threads_started":1}
I20260619 01:07:48.339502 10403 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44331
I20260619 01:07:48.339862 10403 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:48.340647 10403 heartbeater.cc:511] Master 127.0.97.62:44331 requested a full tablet report, sending...
I20260619 01:07:48.343326 10217 ts_manager.cc:194] Registered new tserver with Master: 45e8bdc7bbd64143b89ea257674b63b7 (127.0.97.1:35115)
W20260619 01:07:48.686293 10407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:48.686600 10407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:48.686692 10407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:48.694234 10407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:48.694352 10407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:48.694468 10407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:07:48.703828 10407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:48.705210 10407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:48.707182 10407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:48.721125 10415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:48.721251 10413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:48.721164 10412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:48.721973 10407 server_base.cc:1034] running on GCE node
I20260619 01:07:48.722636 10407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:48.723620 10407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:48.724820 10407 hybrid_clock.cc:648] HybridClock initialized: now 1781831268724767 us; error 56 us; skew 500 ppm
I20260619 01:07:48.725176 10407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:07:48.727566 10407 webserver.cc:460] Webserver started at http://127.0.97.2:34101/ using document root <none> and password file <none>
I20260619 01:07:48.728302 10407 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:48.728454 10407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:48.728778 10407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:48.731894 10407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "b4b72321dbf44900af23d8dfae9168c5"
format_stamp: "Formatted at 2026-06-19 01:07:48 on dist-test-slave-5p8w"
server_key: "434fb02c13fdc94e3ce2fababbc60d01"
server_key_iv: "60385c47dae2e2d98858cbcb23d5bf8c"
server_key_version: "encryptionkey@0"
I20260619 01:07:48.732705 10407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b4b72321dbf44900af23d8dfae9168c5"
format_stamp: "Formatted at 2026-06-19 01:07:48 on dist-test-slave-5p8w"
server_key: "434fb02c13fdc94e3ce2fababbc60d01"
server_key_iv: "60385c47dae2e2d98858cbcb23d5bf8c"
server_key_version: "encryptionkey@0"
I20260619 01:07:48.738327 10407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 01:07:48.742214 10421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:48.743786 10407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:48.743985 10407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "b4b72321dbf44900af23d8dfae9168c5"
format_stamp: "Formatted at 2026-06-19 01:07:48 on dist-test-slave-5p8w"
server_key: "434fb02c13fdc94e3ce2fababbc60d01"
server_key_iv: "60385c47dae2e2d98858cbcb23d5bf8c"
server_key_version: "encryptionkey@0"
I20260619 01:07:48.744239 10407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:48.750404 10407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:48.751353 10407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:48.751646 10407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:48.752628 10407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:48.754259 10407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:48.754393 10407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:48.754523 10407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:48.754607 10407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:48.788976 10407 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40557
I20260619 01:07:48.789036 10533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40557 every 8 connection(s)
I20260619 01:07:48.790441 10407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:07:48.791970 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 10407
I20260619 01:07:48.792186 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:07:48.792667 388 external_mini_cluster.cc:1428] Setting key 69659a0639d7e36416c8d09091ec272b
I20260619 01:07:48.795725 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44331
--builtin_ntp_servers=127.0.97.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 01:07:48.799422 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:48.793231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44439 (local address 127.0.97.62:44331)
0619 01:07:48.793467 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:48.793473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:48.795192 (+ 1719us) server_negotiation.cc:407] Connection header received
0619 01:07:48.797045 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:48.797050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:48.797116 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:48.797239 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:48.798667 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:48.798673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:48.798678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:48.798705 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:48.798737 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:48.798749 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:48.798901 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:48.799087 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 01:07:48.799155 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 01:07:48.800473 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:48.793961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44331 (local address 127.0.97.2:44439)
0619 01:07:48.795041 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:48.795082 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:48.796768 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:48.797517 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:48.797527 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:48.797632 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:48.798342 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:07:48.798345 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:48.798399 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:07:48.798490 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:48.798853 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:48.798866 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:48.798875 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 01:07:48.799172 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 01:07:48.799467 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":351,"threads_started":1}
I20260619 01:07:48.802028 10534 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44331
I20260619 01:07:48.802354 10534 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:48.802994 10534 heartbeater.cc:511] Master 127.0.97.62:44331 requested a full tablet report, sending...
I20260619 01:07:48.804543 10217 ts_manager.cc:194] Registered new tserver with Master: b4b72321dbf44900af23d8dfae9168c5 (127.0.97.2:40557)
W20260619 01:07:49.144588 10537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:49.144929 10537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:49.145089 10537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:49.153915 10537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:49.154064 10537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:49.154271 10537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:07:49.165234 10537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:49.166814 10537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:49.168912 10537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:49.185124 10542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:49.185109 10543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:49.185438 10545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:49.185978 10537 server_base.cc:1034] running on GCE node
I20260619 01:07:49.186755 10537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:49.188413 10537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:49.189721 10537 hybrid_clock.cc:648] HybridClock initialized: now 1781831269189657 us; error 64 us; skew 500 ppm
I20260619 01:07:49.190156 10537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:07:49.192952 10537 webserver.cc:460] Webserver started at http://127.0.97.3:35065/ using document root <none> and password file <none>
I20260619 01:07:49.193882 10537 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:49.194016 10537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:49.194389 10537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:49.197921 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "59fb5543ef4d485d85e575090dc04ddc"
format_stamp: "Formatted at 2026-06-19 01:07:49 on dist-test-slave-5p8w"
server_key: "ff290bb46001f77defb304529a377323"
server_key_iv: "c0fc345e321b76d625f1c442735d7e99"
server_key_version: "encryptionkey@0"
I20260619 01:07:49.198968 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "59fb5543ef4d485d85e575090dc04ddc"
format_stamp: "Formatted at 2026-06-19 01:07:49 on dist-test-slave-5p8w"
server_key: "ff290bb46001f77defb304529a377323"
server_key_iv: "c0fc345e321b76d625f1c442735d7e99"
server_key_version: "encryptionkey@0"
I20260619 01:07:49.204989 10537 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 01:07:49.209276 10551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:49.210988 10537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:49.211202 10537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "59fb5543ef4d485d85e575090dc04ddc"
format_stamp: "Formatted at 2026-06-19 01:07:49 on dist-test-slave-5p8w"
server_key: "ff290bb46001f77defb304529a377323"
server_key_iv: "c0fc345e321b76d625f1c442735d7e99"
server_key_version: "encryptionkey@0"
I20260619 01:07:49.211481 10537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:49.218235 10537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:49.219283 10537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:49.219604 10537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:49.220731 10537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:49.222468 10537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:49.222566 10537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:49.222751 10537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:49.222865 10537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:49.259859 10537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35491
I20260619 01:07:49.259971 10663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35491 every 8 connection(s)
I20260619 01:07:49.261529 10537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:07:49.270586 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 10537
I20260619 01:07:49.270645 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:49.264399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56919 (local address 127.0.97.62:44331)
0619 01:07:49.264619 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:49.264630 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:49.266455 (+ 1825us) server_negotiation.cc:407] Connection header received
0619 01:07:49.268153 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:49.268158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:49.268215 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:49.268321 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:49.269812 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:49.269817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:49.269820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:49.269839 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:49.269871 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:49.269883 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:49.269991 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:49.270227 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 01:07:49.270304 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:07:49.270773 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:07:49.271287 388 external_mini_cluster.cc:1428] Setting key d503219e4a2bdd57c5992e78b01d5909
I20260619 01:07:49.271715 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:49.265159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44331 (local address 127.0.97.3:56919)
0619 01:07:49.266256 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:49.266294 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:49.267867 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:49.268562 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:49.268572 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:49.268677 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:49.269464 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:07:49.269467 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:49.269514 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:07:49.269613 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:49.269985 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:49.269993 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:49.270000 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 01:07:49.270286 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 01:07:49.270524 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":333,"threads_started":1}
I20260619 01:07:49.273404 10664 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44331
I20260619 01:07:49.273775 10664 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:49.274614 10664 heartbeater.cc:511] Master 127.0.97.62:44331 requested a full tablet report, sending...
I20260619 01:07:49.276227 10217 ts_manager.cc:194] Registered new tserver with Master: 59fb5543ef4d485d85e575090dc04ddc (127.0.97.3:35491)
I20260619 01:07:49.287602 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 01:07:49.294520 10405 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:07:49.292909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.0.97.62:44331)
0619 01:07:49.293108 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:49.293113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:49.293619 (+ 506us) server_negotiation.cc:407] Connection header received
0619 01:07:49.293747 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:49.293751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:49.293806 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:49.293904 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:49.294204 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55774: BlockingRecv error: recv got EOF from 127.0.0.1:55774 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:07:49.295184 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 10276
I20260619 01:07:49.298841 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:49.296179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.0.97.1:35115)
0619 01:07:49.296657 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:49.296666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:49.296702 (+ 36us) server_negotiation.cc:407] Connection header received
0619 01:07:49.296763 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:49.296769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:49.297074 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:49.297219 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:49.298192 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:49.298199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:49.298211 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:49.298223 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:49.298268 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:49.298285 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:49.298396 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:49.298514 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 01:07:49.298601 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":233,"threads_started":1}
I20260619 01:07:49.300081 10398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:49.382074 10403 heartbeater.cc:503] Master 127.0.97.62:44331 was elected leader, sending a full tablet report...
I20260619 01:07:49.383085 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 10276
W20260619 01:07:49.400094 10201 connection.cc:535] server connection from 127.0.97.1:34739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 01:07:49.400436 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 10407
I20260619 01:07:49.404007 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:49.401422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.0.97.2:40557)
0619 01:07:49.401914 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:49.401922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:49.401947 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:07:49.402007 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:49.402013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:49.402233 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:49.402380 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:49.403220 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:49.403226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:49.403239 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:49.403257 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:49.403298 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:49.403313 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:49.403433 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:49.403600 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 01:07:49.403714 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":246,"threads_started":1}
I20260619 01:07:49.405241 10529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:49.487558 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 10407
I20260619 01:07:49.504771 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 10537
I20260619 01:07:49.508807 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:49.506018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.0.97.3:35491)
0619 01:07:49.506724 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:49.506739 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:49.506771 (+ 32us) server_negotiation.cc:407] Connection header received
0619 01:07:49.506833 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:49.506840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:49.507080 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:49.507215 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:49.508137 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:49.508143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:49.508155 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:07:49.508177 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:49.508215 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:07:49.508228 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:49.508369 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:49.508431 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 01:07:49.508526 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":420,"threads_started":1}
I20260619 01:07:49.510354 10659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:49.594791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 10537
I20260619 01:07:49.610816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 10186
I20260619 01:07:49.612048 10247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:49.714535 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 10186
2026-06-19T01:07:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2493 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:07:49 dist-test-slave-5p8w krb5kdc[10682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:07:49 dist-test-slave-5p8w krb5kdc[10682](info): set up 2 sockets
Jun 19 01:07:49 dist-test-slave-5p8w krb5kdc[10682](info): commencing operation
krb5kdc: starting...
W20260619 01:07:50.846320 388 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:07:50 dist-test-slave-5p8w krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:07:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:07:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:07:51.084934 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42573
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:51.428761 10695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:51.429162 10695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:51.429397 10695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:51.437119 10695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:07:51.437291 10695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:51.437412 10695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:51.437521 10695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:07:51.437611 10695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:07:51.447836 10695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:51.449440 10695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:51.450819 10695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:51.463476 10703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:51.463572 10701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:51.463537 10700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:51.464511 10695 server_base.cc:1034] running on GCE node
I20260619 01:07:51.465426 10695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:51.466997 10695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:51.468254 10695 hybrid_clock.cc:648] HybridClock initialized: now 1781831271468211 us; error 55 us; skew 500 ppm
I20260619 01:07:51.468670 10695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:51 dist-test-slave-5p8w krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:51.477082 10695 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:07:51.478971 10695 webserver.cc:460] Webserver started at http://127.0.97.62:42359/ using document root <none> and password file <none>
I20260619 01:07:51.479789 10695 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:51.479952 10695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:51.480295 10695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:51.483582 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "fc768712a1ee466fa07fd4e57b784f44"
format_stamp: "Formatted at 2026-06-19 01:07:51 on dist-test-slave-5p8w"
server_key: "709d4ce115ce82ac4621074d336078ac"
server_key_iv: "acde9b0672589d8755fd2fcc1550ba50"
server_key_version: "encryptionkey@0"
I20260619 01:07:51.484458 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "fc768712a1ee466fa07fd4e57b784f44"
format_stamp: "Formatted at 2026-06-19 01:07:51 on dist-test-slave-5p8w"
server_key: "709d4ce115ce82ac4621074d336078ac"
server_key_iv: "acde9b0672589d8755fd2fcc1550ba50"
server_key_version: "encryptionkey@0"
I20260619 01:07:51.490448 10695 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 01:07:51.494234 10710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:51.495728 10695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:51.495950 10695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "fc768712a1ee466fa07fd4e57b784f44"
format_stamp: "Formatted at 2026-06-19 01:07:51 on dist-test-slave-5p8w"
server_key: "709d4ce115ce82ac4621074d336078ac"
server_key_iv: "acde9b0672589d8755fd2fcc1550ba50"
server_key_version: "encryptionkey@0"
I20260619 01:07:51.496189 10695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:51.527398 10695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:51.531952 10695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:51.532277 10695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:51.552403 10695 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42573
I20260619 01:07:51.552425 10761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42573 every 8 connection(s)
I20260619 01:07:51.554414 10695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:07:51.558840 10762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:51.559031 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 10695
I20260619 01:07:51.559228 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:07:51.560019 388 external_mini_cluster.cc:1428] Setting key 5ab766cb3fe4a8866c0b2d67194a5286
I20260619 01:07:51.574117 10762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44: Bootstrap starting.
Jun 19 01:07:51 dist-test-slave-5p8w krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:51.578361 10762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:51.579860 10762 log.cc:824] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:51.583573 10762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44: No bootstrap required, opened a new log
I20260619 01:07:51.587071 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:51.562395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.0.97.62:42573)
0619 01:07:51.563347 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:51.563362 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:51.563413 (+ 51us) server_negotiation.cc:407] Connection header received
0619 01:07:51.565548 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:51.565577 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:51.566111 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:51.566513 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:51.568510 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:51.570435 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:51.572461 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:51.573254 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:51.573487 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:51.578573 (+ 5086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:51.578587 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:51.578619 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:51.578672 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:51.582319 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:51.583321 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:51.583327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:51.583334 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:51.583530 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:51.584015 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:51.584019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:51.584022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:51.584630 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:51.584848 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:51.585435 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 01:07:51.585938 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":445,"threads_started":1}
I20260619 01:07:51.593355 10762 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc768712a1ee466fa07fd4e57b784f44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42573 } }
I20260619 01:07:51.593775 10762 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:51.593883 10762 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc768712a1ee466fa07fd4e57b784f44, State: Initialized, Role: FOLLOWER
I20260619 01:07:51.594571 10762 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc768712a1ee466fa07fd4e57b784f44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42573 } }
I20260619 01:07:51.594810 10762 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:07:51.594941 10762 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:07:51.595134 10762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:51.597667 10762 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc768712a1ee466fa07fd4e57b784f44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42573 } }
I20260619 01:07:51.598266 10762 leader_election.cc:302] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc768712a1ee466fa07fd4e57b784f44; no voters:
I20260619 01:07:51.599037 10762 leader_election.cc:288] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:07:51.599174 10767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:51.600265 10767 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [term 1 LEADER]: Becoming Leader. State: Replica: fc768712a1ee466fa07fd4e57b784f44, State: Running, Role: LEADER
I20260619 01:07:51.600921 10767 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc768712a1ee466fa07fd4e57b784f44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42573 } }
I20260619 01:07:51.601613 10762 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:07:51.604046 10769 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc768712a1ee466fa07fd4e57b784f44. Latest consensus state: current_term: 1 leader_uuid: "fc768712a1ee466fa07fd4e57b784f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc768712a1ee466fa07fd4e57b784f44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42573 } } }
I20260619 01:07:51.604434 10769 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:51.605070 10768 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc768712a1ee466fa07fd4e57b784f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc768712a1ee466fa07fd4e57b784f44" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42573 } } }
I20260619 01:07:51.605484 10768 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44 [sys.catalog]: This master's current role is: LEADER
I20260619 01:07:51.608527 10773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:07:51.614037 10773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:07:51.624167 10773 catalog_manager.cc:1347] Generated new cluster ID: 5683b7ab4f804b788b9cc9bc10b39849
I20260619 01:07:51.624305 10773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:07:51.641594 10773 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:07:51.643011 10773 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:07:51.656215 10773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44: Generated new TSK 0
I20260619 01:07:51.657800 10773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:07:51.717772 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42573
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:52.065868 10789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:52.066183 10789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:52.066278 10789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:52.073953 10789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:52.074079 10789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:52.074194 10789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:07:52.087110 10789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:52.088639 10789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:52.090003 10789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:52.104007 10794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:52.104022 10795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:52.103931 10797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:52.107007 10789 server_base.cc:1034] running on GCE node
I20260619 01:07:52.107758 10789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:52.108927 10789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:52.110188 10789 hybrid_clock.cc:648] HybridClock initialized: now 1781831272110123 us; error 68 us; skew 500 ppm
I20260619 01:07:52.110558 10789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:52 dist-test-slave-5p8w krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:52.113852 10789 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:07:52.115514 10789 webserver.cc:460] Webserver started at http://127.0.97.1:43271/ using document root <none> and password file <none>
I20260619 01:07:52.116297 10789 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:52.116451 10789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:52.116776 10789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:52.119869 10789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "2ef2b66fa4f6495fbbb88ed1dce37370"
format_stamp: "Formatted at 2026-06-19 01:07:52 on dist-test-slave-5p8w"
server_key: "92c410f981284d7ca9bf428fbffac718"
server_key_iv: "1faef4093009da4aa52d4190bc2e097b"
server_key_version: "encryptionkey@0"
I20260619 01:07:52.120683 10789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2ef2b66fa4f6495fbbb88ed1dce37370"
format_stamp: "Formatted at 2026-06-19 01:07:52 on dist-test-slave-5p8w"
server_key: "92c410f981284d7ca9bf428fbffac718"
server_key_iv: "1faef4093009da4aa52d4190bc2e097b"
server_key_version: "encryptionkey@0"
I20260619 01:07:52.126279 10789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 01:07:52.130156 10804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:52.131630 10789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:52.131820 10789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "2ef2b66fa4f6495fbbb88ed1dce37370"
format_stamp: "Formatted at 2026-06-19 01:07:52 on dist-test-slave-5p8w"
server_key: "92c410f981284d7ca9bf428fbffac718"
server_key_iv: "1faef4093009da4aa52d4190bc2e097b"
server_key_version: "encryptionkey@0"
I20260619 01:07:52.132046 10789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:52.162182 10789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:52.166643 10789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:52.166947 10789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:52.167950 10789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:52.169411 10789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:52.169507 10789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:52.169651 10789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:52.169731 10789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:52.203553 10789 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37605
I20260619 01:07:52.203611 10916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37605 every 8 connection(s)
I20260619 01:07:52.204941 10789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:07:52.212802 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 10789
I20260619 01:07:52.213019 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:07:52.213721 388 external_mini_cluster.cc:1428] Setting key b8ee3ad3ab026756839568a595d0ed32
Jun 19 01:07:52 dist-test-slave-5p8w krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:52.226331 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:52.207609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33579 (local address 127.0.97.62:42573)
0619 01:07:52.208037 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:52.208042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:52.209262 (+ 1220us) server_negotiation.cc:407] Connection header received
0619 01:07:52.211119 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:52.211123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:52.211200 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:52.211303 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:52.213001 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:52.213883 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:52.215835 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:52.216222 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:52.216330 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:52.220529 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:52.220534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:52.220538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:52.220570 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:52.223121 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:52.223908 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:52.223918 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:52.223921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:52.224047 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:52.224477 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:52.224484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:52.224488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:52.224762 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:52.224919 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:52.225729 (+ 810us) server_negotiation.cc:299] Negotiation successful
0619 01:07:52.225953 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":219,"threads_started":1}
I20260619 01:07:52.227389 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:52.208228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42573 (local address 127.0.97.1:33579)
0619 01:07:52.209096 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:52.209143 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:52.210867 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:52.211508 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:52.211516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:52.212027 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:52.212803 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:52.212828 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:52.214040 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:52.214045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:52.215695 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:52.215710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:52.216457 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:52.216464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:52.216791 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:52.217601 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:52.217636 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:52.220277 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:52.223301 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:52.223310 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:52.223336 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:52.223770 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:52.224160 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:52.224165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:52.224169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:52.224363 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:52.224892 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:52.224899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:52.225266 (+ 367us) client_negotiation.cc:769] Sending connection context
0619 01:07:52.225635 (+ 369us) client_negotiation.cc:240] Negotiation successful
0619 01:07:52.226201 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":409,"threads_started":1}
I20260619 01:07:52.229084 10917 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42573
I20260619 01:07:52.229439 10917 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:52.230191 10917 heartbeater.cc:511] Master 127.0.97.62:42573 requested a full tablet report, sending...
I20260619 01:07:52.232586 10727 ts_manager.cc:194] Registered new tserver with Master: 2ef2b66fa4f6495fbbb88ed1dce37370 (127.0.97.1:37605)
I20260619 01:07:52.234282 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33579
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:07:52.268867 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42573
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:52.615675 10923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:52.615988 10923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:52.616173 10923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:52.623530 10923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:52.623695 10923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:52.623890 10923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:07:52.634029 10923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:52.635392 10923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:52.636868 10923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:52.650581 10931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:52.650650 10928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:52.650681 10929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:52.651527 10923 server_base.cc:1034] running on GCE node
I20260619 01:07:52.652165 10923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:52.653138 10923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:52.654400 10923 hybrid_clock.cc:648] HybridClock initialized: now 1781831272654328 us; error 61 us; skew 500 ppm
I20260619 01:07:52.654759 10923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:52 dist-test-slave-5p8w krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:52.658119 10923 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:07:52.659770 10923 webserver.cc:460] Webserver started at http://127.0.97.2:35169/ using document root <none> and password file <none>
I20260619 01:07:52.660522 10923 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:52.660668 10923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:52.660979 10923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:52.664101 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "f024f3c5f968475192c792f40e2f019d"
format_stamp: "Formatted at 2026-06-19 01:07:52 on dist-test-slave-5p8w"
server_key: "fd691a0052f5703505f1ef07b4792807"
server_key_iv: "0e431222541f4a9439ba746803501f49"
server_key_version: "encryptionkey@0"
I20260619 01:07:52.664928 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f024f3c5f968475192c792f40e2f019d"
format_stamp: "Formatted at 2026-06-19 01:07:52 on dist-test-slave-5p8w"
server_key: "fd691a0052f5703505f1ef07b4792807"
server_key_iv: "0e431222541f4a9439ba746803501f49"
server_key_version: "encryptionkey@0"
I20260619 01:07:52.670413 10923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 01:07:52.674131 10938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:52.675560 10923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:52.675755 10923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "f024f3c5f968475192c792f40e2f019d"
format_stamp: "Formatted at 2026-06-19 01:07:52 on dist-test-slave-5p8w"
server_key: "fd691a0052f5703505f1ef07b4792807"
server_key_iv: "0e431222541f4a9439ba746803501f49"
server_key_version: "encryptionkey@0"
I20260619 01:07:52.675967 10923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:52.698644 10923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:52.703549 10923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:52.703853 10923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:52.704963 10923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:52.706609 10923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:52.706718 10923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:52.706866 10923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:52.706921 10923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:52.741578 10923 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44675
I20260619 01:07:52.741671 11050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44675 every 8 connection(s)
I20260619 01:07:52.742913 10923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:07:52.752820 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 10923
I20260619 01:07:52.753014 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:07:52.753610 388 external_mini_cluster.cc:1428] Setting key d743302a78df5a1f2fdbc52d9e53022d
Jun 19 01:07:52 dist-test-slave-5p8w krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:52.765578 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:52.745447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41039 (local address 127.0.97.62:42573)
0619 01:07:52.745853 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:52.745858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:52.746978 (+ 1120us) server_negotiation.cc:407] Connection header received
0619 01:07:52.748931 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:52.748936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:52.749044 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:52.749163 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:52.750814 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:52.751534 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:52.753755 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:52.754200 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:52.754311 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:52.758273 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:52.758282 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:52.758290 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:52.758335 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:52.762048 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:52.762890 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:52.762899 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:52.762906 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:52.763069 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:52.763506 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:52.763515 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:52.763521 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:52.763861 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:52.763974 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:52.765030 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0619 01:07:52.765223 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":200,"threads_started":1}
I20260619 01:07:52.767104 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:52.746071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42573 (local address 127.0.97.2:41039)
0619 01:07:52.746798 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:52.746844 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:52.748617 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:52.749449 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:52.749458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:52.749998 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:52.750611 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:52.750633 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:52.751699 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:52.751704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:52.753528 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:52.753544 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:52.754461 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:52.754468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:52.754734 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:52.755514 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:52.755552 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:52.757980 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:52.762233 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:52.762240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:52.762256 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:52.762734 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:52.763184 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:52.763187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:52.763189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:52.763378 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:52.763996 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:52.764007 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:52.764535 (+ 528us) client_negotiation.cc:769] Sending connection context
0619 01:07:52.764896 (+ 361us) client_negotiation.cc:240] Negotiation successful
0619 01:07:52.765560 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":308,"threads_started":1}
I20260619 01:07:52.768657 11051 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42573
I20260619 01:07:52.768993 11051 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:52.769795 11051 heartbeater.cc:511] Master 127.0.97.62:42573 requested a full tablet report, sending...
I20260619 01:07:52.771227 10727 ts_manager.cc:194] Registered new tserver with Master: f024f3c5f968475192c792f40e2f019d (127.0.97.2:44675)
I20260619 01:07:52.772032 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41039
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:07:52.808947 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42573
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:07:53.150403 11057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:07:53.150755 11057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:07:53.150925 11057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:07:53.158646 11057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:07:53.158813 11057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:07:53.159001 11057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:07:53.168947 11057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:07:53.170423 11057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:07:53.171838 11057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:07:53.185876 11065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:53.185895 11062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:07:53.186062 11063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:07:53.186725 11057 server_base.cc:1034] running on GCE node
I20260619 01:07:53.187343 11057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:07:53.188335 11057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:07:53.189543 11057 hybrid_clock.cc:648] HybridClock initialized: now 1781831273189493 us; error 40 us; skew 500 ppm
I20260619 01:07:53.189870 11057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:07:53 dist-test-slave-5p8w krb5kdc[10682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:07:53.193089 11057 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:07:53.194604 11057 webserver.cc:460] Webserver started at http://127.0.97.3:37307/ using document root <none> and password file <none>
I20260619 01:07:53.195335 11057 fs_manager.cc:359] Metadata directory not provided
I20260619 01:07:53.195474 11057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:07:53.195791 11057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:07:53.198913 11057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "ed2066c289ba48b3bccd60b3fb4008ca"
format_stamp: "Formatted at 2026-06-19 01:07:53 on dist-test-slave-5p8w"
server_key: "18d87cb5a46530600e19457e68db5002"
server_key_iv: "d399812ef5e76aef6cf3d1bdcc158901"
server_key_version: "encryptionkey@0"
I20260619 01:07:53.199851 11057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ed2066c289ba48b3bccd60b3fb4008ca"
format_stamp: "Formatted at 2026-06-19 01:07:53 on dist-test-slave-5p8w"
server_key: "18d87cb5a46530600e19457e68db5002"
server_key_iv: "d399812ef5e76aef6cf3d1bdcc158901"
server_key_version: "encryptionkey@0"
I20260619 01:07:53.205981 11057 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260619 01:07:53.210006 11072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:53.211531 11057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:07:53.211731 11057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "ed2066c289ba48b3bccd60b3fb4008ca"
format_stamp: "Formatted at 2026-06-19 01:07:53 on dist-test-slave-5p8w"
server_key: "18d87cb5a46530600e19457e68db5002"
server_key_iv: "d399812ef5e76aef6cf3d1bdcc158901"
server_key_version: "encryptionkey@0"
I20260619 01:07:53.211952 11057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:07:53.237790 10917 heartbeater.cc:503] Master 127.0.97.62:42573 was elected leader, sending a full tablet report...
I20260619 01:07:53.253139 11057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:07:53.257623 11057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:07:53.257952 11057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:07:53.258956 11057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:07:53.260351 11057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:07:53.260480 11057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:53.260622 11057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:07:53.260694 11057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:07:53.294564 11057 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42157
I20260619 01:07:53.294612 11184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42157 every 8 connection(s)
I20260619 01:07:53.295861 11057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:07:53.302814 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 11057
I20260619 01:07:53.303017 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:07:53.303567 388 external_mini_cluster.cc:1428] Setting key 32f2569f8e4f1a4a24336f5442f17a28
Jun 19 01:07:53 dist-test-slave-5p8w krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:07:53.317272 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.298531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33591 (local address 127.0.97.62:42573)
0619 01:07:53.298929 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:53.298933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:53.300056 (+ 1123us) server_negotiation.cc:407] Connection header received
0619 01:07:53.302092 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:53.302096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:53.302181 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:53.302280 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:53.304365 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.305484 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.307456 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.307852 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.307980 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.311540 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:53.311545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:53.311548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:53.311582 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:53.314145 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:53.314919 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:53.314925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:53.314929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:53.315051 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:53.315503 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:53.315509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:53.315513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:53.315797 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:53.315982 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:53.316673 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 01:07:53.316893 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":200,"threads_started":1}
I20260619 01:07:53.318542 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.299134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42573 (local address 127.0.97.3:33591)
0619 01:07:53.299875 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:53.299934 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:53.301804 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:53.302582 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:53.302596 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:53.303357 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:07:53.304186 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.304207 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.305651 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.305655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.307311 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.307323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.308089 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.308097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.308317 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.308980 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:07:53.309002 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:07:53.311316 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:07:53.314326 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:53.314335 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:53.314353 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:53.314775 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:53.315171 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:07:53.315176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:07:53.315180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:07:53.315394 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:07:53.315967 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:07:53.315975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:07:53.316316 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 01:07:53.316587 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 01:07:53.317137 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":269,"threads_started":1}
I20260619 01:07:53.319860 11185 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42573
I20260619 01:07:53.320178 11185 heartbeater.cc:464] Registering TS with master...
I20260619 01:07:53.320905 11185 heartbeater.cc:511] Master 127.0.97.62:42573 requested a full tablet report, sending...
I20260619 01:07:53.322357 10727 ts_manager.cc:194] Registered new tserver with Master: ed2066c289ba48b3bccd60b3fb4008ca (127.0.97.3:42157)
I20260619 01:07:53.323228 10727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33591
I20260619 01:07:53.331755 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:07:53.348203 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.336454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41606 (local address 127.0.97.62:42573)
0619 01:07:53.336657 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:53.336666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:53.337129 (+ 463us) server_negotiation.cc:407] Connection header received
0619 01:07:53.337278 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:53.337282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:53.337406 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:53.337496 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:53.338915 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.339626 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.340798 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.341168 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.341260 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.343581 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:53.343585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:53.343588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:53.343615 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:53.346030 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:53.346638 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:53.346642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:53.346645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:53.346757 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:53.347052 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:53.347055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:53.347057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:53.347349 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:53.347536 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:53.347805 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 01:07:53.347974 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 01:07:53.352483 10727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:07:53.359212 10727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:07:53.385120 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.371163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37605 (local address 127.0.0.1:37194)
0619 01:07:53.371945 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:53.371989 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:53.372162 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:53.373084 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:53.373093 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:53.373129 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:53.373634 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.373651 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.376163 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.376172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.380519 (+ 4347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.380543 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.383711 (+ 3168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.383721 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.384044 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.384118 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 01:07:53.384280 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 01:07:53.384554 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":669,"threads_started":1}
I20260619 01:07:53.386140 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.371330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.0.97.1:37605)
0619 01:07:53.372142 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:53.372150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:53.372179 (+ 29us) server_negotiation.cc:407] Connection header received
0619 01:07:53.372385 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:53.372399 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:53.372605 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:53.372800 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:53.373883 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.375952 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.380799 (+ 4847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.383020 (+ 2221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.383194 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.383528 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:53.385670 (+ 2142us) server_negotiation.cc:299] Negotiation successful
0619 01:07:53.385807 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":666,"threads_started":1}
I20260619 01:07:53.390408 10852 tablet_service.cc:1467] Processing CreateTablet for tablet cd501beabb054e0990c1866792915762 (DEFAULT_TABLE table=test-table [id=95702bd2671e41f8ba64f781856d7042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:53.392060 10852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd501beabb054e0990c1866792915762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:53.392596 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.371858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.0.97.2:44675)
0619 01:07:53.382928 (+ 11070us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:53.382942 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:53.383917 (+ 975us) server_negotiation.cc:407] Connection header received
0619 01:07:53.384043 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:53.384051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:53.384425 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:53.384639 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:53.386471 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.388235 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.390217 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.391589 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.391731 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.391929 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:53.392134 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 01:07:53.392287 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10837,"thread_start_us":11088,"threads_started":1}
I20260619 01:07:53.392637 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.371736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44675 (local address 127.0.0.1:58194)
0619 01:07:53.383606 (+ 11870us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:53.383631 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:53.383774 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:53.385868 (+ 2094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:53.385875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:53.385900 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:53.386240 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.386253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.388386 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.388392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.390067 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.390079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.391774 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.391780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.391938 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.391977 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 01:07:53.392082 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:07:53.392207 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11731,"mutex_wait_us":86,"thread_start_us":11597,"threads_started":1}
I20260619 01:07:53.393468 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.372781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42157 (local address 127.0.0.1:34536)
0619 01:07:53.385685 (+ 12904us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:53.385704 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:53.385815 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:53.386548 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:53.386554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:53.386578 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:53.386950 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.386962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.388562 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.388567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.390682 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.390695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.392738 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.392746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.392903 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.392933 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 01:07:53.393019 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 01:07:53.393146 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12792,"thread_start_us":12284,"threads_started":1}
I20260619 01:07:53.393903 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.373019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.0.97.3:42157)
0619 01:07:53.385741 (+ 12722us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:53.385752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:53.385783 (+ 31us) server_negotiation.cc:407] Connection header received
0619 01:07:53.386002 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:53.386013 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:53.386259 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:53.386433 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:53.387158 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.388386 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.390861 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.392233 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.392353 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.392573 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:53.393414 (+ 841us) server_negotiation.cc:299] Negotiation successful
0619 01:07:53.393561 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12548,"thread_start_us":12390,"threads_started":1}
I20260619 01:07:53.396368 10986 tablet_service.cc:1467] Processing CreateTablet for tablet cd501beabb054e0990c1866792915762 (DEFAULT_TABLE table=test-table [id=95702bd2671e41f8ba64f781856d7042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:53.397573 10986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd501beabb054e0990c1866792915762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:53.397481 11120 tablet_service.cc:1467] Processing CreateTablet for tablet cd501beabb054e0990c1866792915762 (DEFAULT_TABLE table=test-table [id=95702bd2671e41f8ba64f781856d7042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:07:53.398887 11120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd501beabb054e0990c1866792915762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:07:53.407473 11204 tablet_bootstrap.cc:492] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: Bootstrap starting.
I20260619 01:07:53.410169 11205 tablet_bootstrap.cc:492] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: Bootstrap starting.
I20260619 01:07:53.411398 11204 tablet_bootstrap.cc:654] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:53.412412 11204 log.cc:824] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:53.414108 11205 tablet_bootstrap.cc:654] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:53.415802 11205 log.cc:824] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:53.420192 11207 tablet_bootstrap.cc:492] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: Bootstrap starting.
I20260619 01:07:53.421948 11204 tablet_bootstrap.cc:492] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: No bootstrap required, opened a new log
I20260619 01:07:53.422160 11205 tablet_bootstrap.cc:492] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: No bootstrap required, opened a new log
I20260619 01:07:53.422271 11204 ts_tablet_manager.cc:1397] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260619 01:07:53.422477 11205 ts_tablet_manager.cc:1397] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20260619 01:07:53.424120 11207 tablet_bootstrap.cc:654] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: Neither blocks nor log segments found. Creating new log.
I20260619 01:07:53.425486 11207 log.cc:824] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: Log is configured to *not* fsync() on all Append() calls
I20260619 01:07:53.429019 11207 tablet_bootstrap.cc:492] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: No bootstrap required, opened a new log
I20260619 01:07:53.429386 11207 ts_tablet_manager.cc:1397] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260619 01:07:53.430699 11205 raft_consensus.cc:348] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37605 } } peers { permanent_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } } peers { permanent_uuid: "f024f3c5f968475192c792f40e2f019d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44675 } }
I20260619 01:07:53.431102 11205 raft_consensus.cc:374] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:53.431236 11205 raft_consensus.cc:729] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f024f3c5f968475192c792f40e2f019d, State: Initialized, Role: FOLLOWER
I20260619 01:07:53.431870 11205 consensus_queue.cc:260] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37605 } } peers { permanent_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } } peers { permanent_uuid: "f024f3c5f968475192c792f40e2f019d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44675 } }
I20260619 01:07:53.432979 11051 heartbeater.cc:503] Master 127.0.97.62:42573 was elected leader, sending a full tablet report...
I20260619 01:07:53.432860 11204 raft_consensus.cc:348] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37605 } } peers { permanent_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } } peers { permanent_uuid: "f024f3c5f968475192c792f40e2f019d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44675 } }
I20260619 01:07:53.433333 11204 raft_consensus.cc:374] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:53.433429 11204 raft_consensus.cc:729] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef2b66fa4f6495fbbb88ed1dce37370, State: Initialized, Role: FOLLOWER
I20260619 01:07:53.434209 11204 consensus_queue.cc:260] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37605 } } peers { permanent_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } } peers { permanent_uuid: "f024f3c5f968475192c792f40e2f019d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44675 } }
I20260619 01:07:53.437343 11205 ts_tablet_manager.cc:1428] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s
I20260619 01:07:53.438902 11204 ts_tablet_manager.cc:1428] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260619 01:07:53.441390 11207 raft_consensus.cc:348] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37605 } } peers { permanent_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } } peers { permanent_uuid: "f024f3c5f968475192c792f40e2f019d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44675 } }
I20260619 01:07:53.441756 11207 raft_consensus.cc:374] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:07:53.441854 11207 raft_consensus.cc:729] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2066c289ba48b3bccd60b3fb4008ca, State: Initialized, Role: FOLLOWER
I20260619 01:07:53.442471 11207 consensus_queue.cc:260] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37605 } } peers { permanent_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } } peers { permanent_uuid: "f024f3c5f968475192c792f40e2f019d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44675 } }
I20260619 01:07:53.443403 11185 heartbeater.cc:503] Master 127.0.97.62:42573 was elected leader, sending a full tablet report...
I20260619 01:07:53.444036 11207 ts_tablet_manager.cc:1428] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20260619 01:07:53.457541 10918 tablet.cc:2369] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:07:53.495007 11052 tablet.cc:2369] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:07:53.547556 11186 tablet.cc:2369] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:07:53.664441 11212 raft_consensus.cc:482] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:07:53.664836 11212 raft_consensus.cc:504] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37605 } } peers { permanent_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } } peers { permanent_uuid: "f024f3c5f968475192c792f40e2f019d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44675 } }
I20260619 01:07:53.666916 11212 leader_election.cc:288] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ef2b66fa4f6495fbbb88ed1dce37370 (127.0.97.1:37605), f024f3c5f968475192c792f40e2f019d (127.0.97.2:44675)
I20260619 01:07:53.674546 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.667386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49719 (local address 127.0.97.1:37605)
0619 01:07:53.667619 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:53.667628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:53.667656 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:07:53.667947 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:53.667954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:53.668041 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:53.668147 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:53.668664 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.669823 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.672262 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.673539 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.673691 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.673829 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:53.674150 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 01:07:53.674282 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 01:07:53.674551 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.667842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34877 (local address 127.0.97.2:44675)
0619 01:07:53.668045 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:53.668052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:53.668257 (+ 205us) server_negotiation.cc:407] Connection header received
0619 01:07:53.668398 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:53.668402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:53.668459 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:53.668549 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:07:53.669069 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.670099 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.672414 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.673683 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.673807 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.673904 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:53.674203 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 01:07:53.674325 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 01:07:53.674657 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.667332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37605 (local address 127.0.97.3:49719)
0619 01:07:53.667556 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:53.667578 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:53.667725 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:53.668175 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:53.668183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:53.668220 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:53.668473 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.668482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.669967 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.669971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.672066 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.672082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.673697 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.673702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.673935 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.673977 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 01:07:53.674099 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 01:07:53.674253 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"spinlock_wait_cycles":27008}
I20260619 01:07:53.674679 11213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.667392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44675 (local address 127.0.97.3:34877)
0619 01:07:53.668139 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 01:07:53.668154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:07:53.668266 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:07:53.668600 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:07:53.668604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:07:53.668631 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:07:53.668909 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.668916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.670232 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.670237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.672264 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:07:53.672273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.673882 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.673888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:07:53.674044 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.674085 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 01:07:53.674162 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 01:07:53.674262 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":462,"threads_started":1}
I20260619 01:07:53.675784 11006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd501beabb054e0990c1866792915762" candidate_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f024f3c5f968475192c792f40e2f019d" is_pre_election: true
I20260619 01:07:53.675784 10872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd501beabb054e0990c1866792915762" candidate_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" is_pre_election: true
I20260619 01:07:53.676343 10872 raft_consensus.cc:2454] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2066c289ba48b3bccd60b3fb4008ca in term 0.
I20260619 01:07:53.676344 11006 raft_consensus.cc:2454] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2066c289ba48b3bccd60b3fb4008ca in term 0.
I20260619 01:07:53.677100 11073 leader_election.cc:302] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ef2b66fa4f6495fbbb88ed1dce37370, ed2066c289ba48b3bccd60b3fb4008ca, f024f3c5f968475192c792f40e2f019d; no voters:
I20260619 01:07:53.677646 11212 raft_consensus.cc:2781] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:07:53.677798 11212 raft_consensus.cc:482] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:07:53.677904 11212 raft_consensus.cc:3037] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:53.680207 11212 raft_consensus.cc:504] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37605 } } peers { permanent_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } } peers { permanent_uuid: "f024f3c5f968475192c792f40e2f019d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44675 } }
I20260619 01:07:53.680883 11212 leader_election.cc:288] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [CANDIDATE]: Term 1 election: Requested vote from peers 2ef2b66fa4f6495fbbb88ed1dce37370 (127.0.97.1:37605), f024f3c5f968475192c792f40e2f019d (127.0.97.2:44675)
I20260619 01:07:53.681419 10872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd501beabb054e0990c1866792915762" candidate_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370"
I20260619 01:07:53.681445 11006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd501beabb054e0990c1866792915762" candidate_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f024f3c5f968475192c792f40e2f019d"
I20260619 01:07:53.681660 11006 raft_consensus.cc:3037] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:53.681643 10872 raft_consensus.cc:3037] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:07:53.683854 11006 raft_consensus.cc:2454] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2066c289ba48b3bccd60b3fb4008ca in term 1.
I20260619 01:07:53.683856 10872 raft_consensus.cc:2454] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2066c289ba48b3bccd60b3fb4008ca in term 1.
I20260619 01:07:53.684361 11073 leader_election.cc:302] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ef2b66fa4f6495fbbb88ed1dce37370, ed2066c289ba48b3bccd60b3fb4008ca, f024f3c5f968475192c792f40e2f019d; no voters:
I20260619 01:07:53.684718 11212 raft_consensus.cc:2781] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:07:53.685148 11212 raft_consensus.cc:686] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 1 LEADER]: Becoming Leader. State: Replica: ed2066c289ba48b3bccd60b3fb4008ca, State: Running, Role: LEADER
I20260619 01:07:53.685657 11212 consensus_queue.cc:237] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37605 } } peers { permanent_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } } peers { permanent_uuid: "f024f3c5f968475192c792f40e2f019d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44675 } }
I20260619 01:07:53.690228 10727 catalog_manager.cc:5495] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca reported cstate change: term changed from 0 to 1, leader changed from <none> to ed2066c289ba48b3bccd60b3fb4008ca (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2066c289ba48b3bccd60b3fb4008ca" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f024f3c5f968475192c792f40e2f019d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44675 } health_report { overall_health: UNKNOWN } } }
I20260619 01:07:53.760296 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.755226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.0.97.3:42157)
0619 01:07:53.755454 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:53.755459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:53.755494 (+ 35us) server_negotiation.cc:407] Connection header received
0619 01:07:53.755703 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:53.755706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:53.755791 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:53.755915 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:07:53.756265 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.756942 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.758290 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.758679 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.758771 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.759213 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:07:53.759717 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:07:53.759824 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:53.759921 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 01:07:53.759985 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 01:07:53.768726 10872 raft_consensus.cc:1261] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370 [term 1 FOLLOWER]: Refusing update from remote peer ed2066c289ba48b3bccd60b3fb4008ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:53.768726 11006 raft_consensus.cc:1261] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d [term 1 FOLLOWER]: Refusing update from remote peer ed2066c289ba48b3bccd60b3fb4008ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:07:53.769747 11212 consensus_queue.cc:1035] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef2b66fa4f6495fbbb88ed1dce37370" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:53.770077 11214 consensus_queue.cc:1035] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f024f3c5f968475192c792f40e2f019d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:07:53.783640 11218 mvcc.cc:205] Tried to move back new op lower bound from 7298380897345638400 to 7298380897020076032. Current Snapshot: MvccSnapshot[applied={T|T < 7298380897345638400}]
I20260619 01:07:53.797994 10726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:53.798318 10726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:07:53.802250 10726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc768712a1ee466fa07fd4e57b784f44: Sending DeleteTablet for 3 replicas of tablet cd501beabb054e0990c1866792915762
I20260619 01:07:53.803638 11120 tablet_service.cc:1514] Processing DeleteTablet for tablet cd501beabb054e0990c1866792915762 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34536
I20260619 01:07:53.803663 10986 tablet_service.cc:1514] Processing DeleteTablet for tablet cd501beabb054e0990c1866792915762 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58194
I20260619 01:07:53.803682 10852 tablet_service.cc:1514] Processing DeleteTablet for tablet cd501beabb054e0990c1866792915762 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:07:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37194
I20260619 01:07:53.804666 11227 tablet_replica.cc:330] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: stopping tablet replica
I20260619 01:07:53.805208 11229 tablet_replica.cc:330] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: stopping tablet replica
I20260619 01:07:53.805784 11229 raft_consensus.cc:2229] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:53.806365 11227 raft_consensus.cc:2229] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:07:53.806394 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 10789
I20260619 01:07:53.807010 11229 raft_consensus.cc:2258] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:53.807708 11227 raft_consensus.cc:2258] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:53.808806 11228 tablet_replica.cc:330] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: stopping tablet replica
I20260619 01:07:53.809230 11228 raft_consensus.cc:2229] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:07:53.809785 11228 raft_consensus.cc:2258] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:07:53.811127 11227 ts_tablet_manager.cc:1905] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:53.811823 11228 ts_tablet_manager.cc:1905] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:53.812068 11229 ts_tablet_manager.cc:1905] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:07:53.818652 11228 ts_tablet_manager.cc:1918] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:53.818828 11228 log.cc:1196] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/cd501beabb054e0990c1866792915762
I20260619 01:07:53.819257 11227 ts_tablet_manager.cc:1918] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:53.819424 11228 ts_tablet_manager.cc:1939] T cd501beabb054e0990c1866792915762 P 2ef2b66fa4f6495fbbb88ed1dce37370: Deleting consensus metadata
I20260619 01:07:53.819439 11227 log.cc:1196] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/cd501beabb054e0990c1866792915762
I20260619 01:07:53.819952 11227 ts_tablet_manager.cc:1939] T cd501beabb054e0990c1866792915762 P ed2066c289ba48b3bccd60b3fb4008ca: Deleting consensus metadata
I20260619 01:07:53.821542 10713 catalog_manager.cc:4841] TS 2ef2b66fa4f6495fbbb88ed1dce37370 (127.0.97.1:37605): tablet cd501beabb054e0990c1866792915762 (table test-table [id=95702bd2671e41f8ba64f781856d7042]) successfully deleted
I20260619 01:07:53.821911 10713 catalog_manager.cc:4841] TS ed2066c289ba48b3bccd60b3fb4008ca (127.0.97.3:42157): tablet cd501beabb054e0990c1866792915762 (table test-table [id=95702bd2671e41f8ba64f781856d7042]) successfully deleted
I20260619 01:07:53.822108 11229 ts_tablet_manager.cc:1918] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:07:53.822291 11229 log.cc:1196] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/cd501beabb054e0990c1866792915762
I20260619 01:07:53.822757 11229 ts_tablet_manager.cc:1939] T cd501beabb054e0990c1866792915762 P f024f3c5f968475192c792f40e2f019d: Deleting consensus metadata
I20260619 01:07:53.824616 10711 catalog_manager.cc:4841] TS f024f3c5f968475192c792f40e2f019d (127.0.97.2:44675): tablet cd501beabb054e0990c1866792915762 (table test-table [id=95702bd2671e41f8ba64f781856d7042]) successfully deleted
Jun 19 01:07:53 dist-test-slave-5p8w krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:07:53.837513 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.808413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.0.97.1:37605)
0619 01:07:53.808837 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:53.808843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:53.817935 (+ 9092us) server_negotiation.cc:407] Connection header received
0619 01:07:53.818147 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:53.818152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:53.818235 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:53.818364 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:53.823049 (+ 4685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.824119 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.826040 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.826442 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.826547 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.831448 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:53.831463 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:53.831480 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:53.831520 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:53.834552 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:53.835348 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:53.835356 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:53.835359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:53.835499 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:53.835921 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:53.835925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:53.835927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:53.836278 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:53.836597 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:53.836922 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 01:07:53.837149 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260619 01:07:53.838214 10912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:53.931308 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 10789
I20260619 01:07:53.956048 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 10923
Jun 19 01:07:53 dist-test-slave-5p8w krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:07:53.973393 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:53.956856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.0.97.2:44675)
0619 01:07:53.957040 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:53.957046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:53.957067 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:07:53.957135 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:53.957143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:53.957235 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:53.957403 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:53.958626 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.959756 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.961097 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:53.961780 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:53.961907 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:53.966186 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:53.966209 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:53.966231 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:53.966280 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:53.970393 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:53.971188 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:53.971198 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:53.971201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:53.971349 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:53.971705 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:53.971710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:53.971714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:53.972195 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:53.972458 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:53.972724 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 01:07:53.972955 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:07:53.974156 11046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:54.064342 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 10923
I20260619 01:07:54.082366 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 11057
Jun 19 01:07:54 dist-test-slave-5p8w krb5kdc[10682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:07:54.100647 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:07:54.083195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.0.97.3:42157)
0619 01:07:54.083392 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 01:07:54.083399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:07:54.083423 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:07:54.083582 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:07:54.083587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:07:54.083678 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:07:54.083799 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:07:54.085102 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:54.086351 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:54.087699 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:07:54.088395 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:07:54.088573 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:07:54.093072 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:07:54.093101 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:07:54.093124 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:07:54.093173 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:07:54.097248 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:54.098131 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:54.098142 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:54.098145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:54.098377 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:07:54.098777 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:07:54.098788 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:07:54.098793 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:07:54.099284 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:07:54.099633 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 01:07:54.099845 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 01:07:54.100113 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 01:07:54.101574 11180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:54.225538 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 11057
I20260619 01:07:54.245096 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 10695
I20260619 01:07:54.246446 10757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:07:54.357242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 10695
2026-06-19T01:07:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4645 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:07:54 dist-test-slave-5p8w krb5kdc[11237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:07:54 dist-test-slave-5p8w krb5kdc[11237](info): set up 2 sockets
Jun 19 01:07:54 dist-test-slave-5p8w krb5kdc[11237](info): commencing operation
krb5kdc: starting...
W20260619 01:07:55.478538 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:07:55 dist-test-slave-5p8w krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:07:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 01:07:55.827929 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbdbgje/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/postgres -l logfile start
2026-06-19 01:07:58.635 UTC [11267] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 01:07:58.635 UTC [11267] LOG: listening on IPv4 address "127.0.97.20", port 45701
2026-06-19 01:07:58.638 UTC [11267] LOG: listening on Unix socket "/tmp/.s.PGSQL.45701"
2026-06-19 01:07:58.661 UTC [11270] LOG: database system was shut down at 2026-06-19 01:07:56 UTC
2026-06-19 01:07:58.665 UTC [11267] LOG: database system is ready to accept connections
I20260619 01:07:59.657900 388 mini_postgres.cc:96] Postgres bound to 45701
2026-06-19 01:07:59.676 UTC [11278] FATAL: database "slave" does not exist
127.0.97.20:45701 - accepting connections
I20260619 01:07:59.677584 388 mini_ranger.cc:161] Starting Ranger...
I20260619 01:07:59.706080 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 01:08:00.246524 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 01:08:00.246722 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-admin
2026-06-19 01:08:00,541 [I] DB FLAVOR :POSTGRES
2026-06-19 01:08:00,541 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 01:08:00,542 [I] Checking connection..
2026-06-19 01:08:00,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 01:08:00,826 [I] Checking connection passed.
2026-06-19 01:08:00,826 [I] --------- Verifying version history table ---------
2026-06-19 01:08:00,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:08:01,091 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 01:08:01,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:08:01,345 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 01:08:01,345 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 01:08:01,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 01:08:01.599 UTC [11342] WARNING: there is no transaction in progress
2026-06-19 01:08:01,606 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 01:08:01,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:08:01,889 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 01:08:01,890 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 01:08:01,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 01:08:02,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 01:08:02,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5p8w.c.gcp-upstream.internal','N') ;"
2026-06-19 01:08:02,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 01:08:02,948 [I] Table x_portal_user does not exist in database ranger
2026-06-19 01:08:02,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 01:08:03,206 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 01:08:03,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 01:08:03,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 01:08:03,699 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 01:08:03,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 01:08:04.794 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:04.815 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:04.834 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:04.852 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:04.872 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:04.893 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:04.956 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:04.965 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:04.983 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:04.996 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:05.009 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:05.026 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:05.038 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:05.046 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:05.054 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:05.063 UTC [11450] WARNING: there is no transaction in progress
2026-06-19 01:08:05,864 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 01:08:05,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 01:08:06,113 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 01:08:06,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 01:08:06,380 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 01:08:06,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 01:08:06,658 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 01:08:06,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 01:08:06,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 01:08:07,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5p8w.c.gcp-upstream.internal';"
2026-06-19 01:08:07,442 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 01:08:07,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 01:08:07,689 [I] DB_PATCHES have already been applied
I20260619 01:08:07.695514 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbdbgje/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 01:08:07.695674 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 01:08:07.711892 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:42271
Jun 19, 2026 1:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 1:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 1:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 1:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 1:08:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 01:08:08.876211 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.199s user 0.003s sys 0.042s
Jun 19, 2026 1:08:30 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 1:08:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 1:08:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 1:08:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 1:08:31 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 01:08:32.592440 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 01:08:32 dist-test-slave-5p8w krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831312, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 01:08:32.775193 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 01:08:32.797559 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 01:08:33.719264 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 01:08:33.719408 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-kms
I20260619 01:08:33.719563 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:45701
2026-06-19 01:08:34,149 [I] DB FLAVOR :POSTGRES
2026-06-19 01:08:34,150 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 01:08:34,150 [I] Checking connection
2026-06-19 01:08:34,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 01:08:34,417 [I] connection success
2026-06-19 01:08:34,417 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 01:08:34,417 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 01:08:34,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 01:08:34,698 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 01:08:34,699 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 01:08:34,699 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 01:08:34,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:45701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 01:08:34,960 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 01:08:35.424643 388 mini_ranger_kms.cc:310] Created kms service
I20260619 01:08:35.606731 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 01:08:35.652601 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 01:08:35 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831312, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:08:36.112516 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 01:08:36.112850 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 01:08:36.112913 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 01:08:36.131429 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11631
I20260619 01:08:36.131644 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:49921
Jun 19, 2026 1:08:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:08:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:08:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 1:08:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 1:08:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 1:08:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 01:08:38 dist-test-slave-5p8w krb5kdc[11237](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781831318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:08:38.544385 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.769s user 0.120s sys 0.327s
I20260619 01:08:38.544651 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 01:08:38.544858 388 mini_ranger_kms.cc:401] 127.0.97.20:49921/kms/v1/keys
Jun 19 01:08:38 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 01:08:39 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 01:08:41 dist-test-slave-5p8w krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:08:41.706072 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37277
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37277
--ranger_config_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49921/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:08:42.072144 11681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:08:42.072530 11681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:08:42.072700 11681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:08:42.080209 11681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:08:42.080389 11681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:08:42.080509 11681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:08:42.080610 11681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:08:42.080699 11681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:08:42.091414 11681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:49921/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37277
--ranger_config_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:08:42.092823 11681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:08:42.094204 11681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 01:08:42.108131 11681 server_base.cc:1034] running on GCE node
W20260619 01:08:42.108093 11687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:08:42.108178 11686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:08:42.108093 11689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:08:42.109658 11681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:08:42.111196 11681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:08:42.112421 11681 hybrid_clock.cc:648] HybridClock initialized: now 1781831322112381 us; error 42 us; skew 500 ppm
I20260619 01:08:42.112821 11681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:08:42 dist-test-slave-5p8w krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:08:42.116267 11681 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:08:42.117913 11681 webserver.cc:460] Webserver started at http://127.0.97.62:43865/ using document root <none> and password file <none>
I20260619 01:08:42.118592 11681 fs_manager.cc:359] Metadata directory not provided
I20260619 01:08:42.118688 11681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:08:42.118953 11681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:08:42 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:08:42.186882 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "55fa0202bc2d422e9e27ea5dea98c177"
format_stamp: "Formatted at 2026-06-19 01:08:42 on dist-test-slave-5p8w"
server_key: "e6fc0938ef1bfc1ec7dda8646f1847fc"
server_key_iv: "ced599654c0154e79329fc6a4619a5c3"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:42.187762 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "55fa0202bc2d422e9e27ea5dea98c177"
format_stamp: "Formatted at 2026-06-19 01:08:42 on dist-test-slave-5p8w"
server_key: "e6fc0938ef1bfc1ec7dda8646f1847fc"
server_key_iv: "ced599654c0154e79329fc6a4619a5c3"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:42.193796 11681 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260619 01:08:42.217425 11696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:08:42.219187 11681 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260619 01:08:42.219434 11681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "55fa0202bc2d422e9e27ea5dea98c177"
format_stamp: "Formatted at 2026-06-19 01:08:42 on dist-test-slave-5p8w"
server_key: "e6fc0938ef1bfc1ec7dda8646f1847fc"
server_key_iv: "ced599654c0154e79329fc6a4619a5c3"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:42.219691 11681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:08:42.241060 11681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:08:42.245589 11681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:08:42.245908 11681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:08:42.268292 11681 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37277
I20260619 01:08:42.268316 11747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37277 every 8 connection(s)
I20260619 01:08:42.269879 11681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:08:42.274715 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:08:42.278146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 11681
I20260619 01:08:42.278425 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:08:42.279088 388 external_mini_cluster.cc:1428] Setting key ccd62312c531d634edf7824e45326dd6
I20260619 01:08:42.289350 11748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177: Bootstrap starting.
I20260619 01:08:42.293032 11748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177: Neither blocks nor log segments found. Creating new log.
I20260619 01:08:42.294127 11748 log.cc:824] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177: Log is configured to *not* fsync() on all Append() calls
Jun 19 01:08:42 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:08:42.297492 11748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177: No bootstrap required, opened a new log
I20260619 01:08:42.304689 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:42.281305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.0.97.62:37277)
0619 01:08:42.282223 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:42.282241 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:42.282296 (+ 55us) server_negotiation.cc:407] Connection header received
0619 01:08:42.284292 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:42.284331 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:42.285319 (+ 988us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:42.285877 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:08:42.287053 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:42.288985 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:42.290450 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:42.291621 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:42.291796 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:42.295943 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:08:42.295953 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:08:42.295970 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:08:42.296005 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:08:42.299819 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:42.300710 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:42.300718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:42.300725 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:42.301012 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:42.301526 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:42.301531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:42.301535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:42.302230 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:08:42.302595 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:42.303108 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 01:08:42.303570 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":386,"threads_started":1}
I20260619 01:08:42.306811 11748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fa0202bc2d422e9e27ea5dea98c177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } }
I20260619 01:08:42.307268 11748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:08:42.307372 11748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fa0202bc2d422e9e27ea5dea98c177, State: Initialized, Role: FOLLOWER
I20260619 01:08:42.308080 11748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fa0202bc2d422e9e27ea5dea98c177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } }
I20260619 01:08:42.308359 11748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:08:42.308463 11748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:08:42.308591 11748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:08:42.310813 11748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fa0202bc2d422e9e27ea5dea98c177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } }
I20260619 01:08:42.311255 11748 leader_election.cc:302] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55fa0202bc2d422e9e27ea5dea98c177; no voters:
I20260619 01:08:42.311817 11748 leader_election.cc:288] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:08:42.311962 11753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:08:42.312659 11753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [term 1 LEADER]: Becoming Leader. State: Replica: 55fa0202bc2d422e9e27ea5dea98c177, State: Running, Role: LEADER
I20260619 01:08:42.316114 11753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fa0202bc2d422e9e27ea5dea98c177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } }
I20260619 01:08:42.316857 11748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:08:42.319239 11755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55fa0202bc2d422e9e27ea5dea98c177. Latest consensus state: current_term: 1 leader_uuid: "55fa0202bc2d422e9e27ea5dea98c177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fa0202bc2d422e9e27ea5dea98c177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } } }
I20260619 01:08:42.319326 11754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55fa0202bc2d422e9e27ea5dea98c177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fa0202bc2d422e9e27ea5dea98c177" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37277 } } }
I20260619 01:08:42.319793 11754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [sys.catalog]: This master's current role is: LEADER
I20260619 01:08:42.321993 11755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177 [sys.catalog]: This master's current role is: LEADER
I20260619 01:08:42.323063 11760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:08:42.329093 11760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:08:42.330363 11748 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 01:08:42.348565 11760 catalog_manager.cc:1347] Generated new cluster ID: 8bb95d7d7ec54ba29a94bbfb60e999e0
I20260619 01:08:42.348789 11760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:08:42.363947 11760 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:08:42.366086 11760 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:08:42.390084 11760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177: Generated new TSK 0
I20260619 01:08:42.391403 11760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:08:43.171475 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:08:43 dist-test-slave-5p8w krb5kdc[11237](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781831323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 01:08:43.660007 11793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:08:43.660459 11793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:08:43.660687 11793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:08:43.672915 11793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:08:43.673095 11793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:08:43.673391 11793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:08:43.690733 11793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:49921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:08:43.692835 11793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:08:43.694777 11793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:08:43.718174 11803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:08:43.719622 11804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:08:43.727123 11806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:08:43.727625 11793 server_base.cc:1034] running on GCE node
I20260619 01:08:43.728328 11793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:08:43.729503 11793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:08:43.730983 11793 hybrid_clock.cc:648] HybridClock initialized: now 1781831323730916 us; error 62 us; skew 500 ppm
I20260619 01:08:43.731367 11793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:08:43 dist-test-slave-5p8w krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:08:43.735009 11793 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:08:43.737083 11793 webserver.cc:460] Webserver started at http://127.0.97.1:46741/ using document root <none> and password file <none>
I20260619 01:08:43.737926 11793 fs_manager.cc:359] Metadata directory not provided
I20260619 01:08:43.738087 11793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:08:43.738438 11793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:08:43 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:08:43.758140 11793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "6c9d3f0ad0c44b0f892736eeb1859a13"
format_stamp: "Formatted at 2026-06-19 01:08:43 on dist-test-slave-5p8w"
server_key: "acf745920b02f442d3734c9d3b19c693"
server_key_iv: "2b033e951e1694ce0ea1144de10c50bc"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:43.759621 11793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6c9d3f0ad0c44b0f892736eeb1859a13"
format_stamp: "Formatted at 2026-06-19 01:08:43 on dist-test-slave-5p8w"
server_key: "acf745920b02f442d3734c9d3b19c693"
server_key_iv: "2b033e951e1694ce0ea1144de10c50bc"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:43.774360 11793 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.008s
I20260619 01:08:43.808537 11813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:08:43.811122 11793 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.002s
I20260619 01:08:43.811393 11793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "6c9d3f0ad0c44b0f892736eeb1859a13"
format_stamp: "Formatted at 2026-06-19 01:08:43 on dist-test-slave-5p8w"
server_key: "acf745920b02f442d3734c9d3b19c693"
server_key_iv: "2b033e951e1694ce0ea1144de10c50bc"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:43.811894 11793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:08:43.879344 11793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:08:43.885321 11793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:08:43.885711 11793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:08:43.888089 11793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:08:43.890143 11793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:08:43.890256 11793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:08:43.890437 11793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:08:43.890506 11793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:08:43 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:08:44.068543 11793 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42587
I20260619 01:08:44.068544 11925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42587 every 8 connection(s)
I20260619 01:08:44.070154 11793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:08:44.077409 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 11793
I20260619 01:08:44.077665 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:08:44.078735 388 external_mini_cluster.cc:1428] Setting key 86dd6fb82128de68f95966b71133ecb9
Jun 19 01:08:44 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:08:44.121902 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:44.075385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40617 (local address 127.0.97.62:37277)
0619 01:08:44.095621 (+ 20236us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:44.095632 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:44.100830 (+ 5198us) server_negotiation.cc:407] Connection header received
0619 01:08:44.101963 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:44.101977 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:44.102087 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:44.102218 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:08:44.103858 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:44.104828 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:44.106708 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:44.107363 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:44.107508 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:44.112628 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:08:44.112636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:08:44.112641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:08:44.112688 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:08:44.116873 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:44.118232 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:44.118238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:44.118243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:44.118486 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:44.119086 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:44.119092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:44.119096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:44.119475 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:08:44.119708 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:44.121201 (+ 1493us) server_negotiation.cc:299] Negotiation successful
0619 01:08:44.121503 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20037,"thread_start_us":19869,"threads_started":1}
I20260619 01:08:44.123931 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:44.074526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37277 (local address 127.0.97.1:40617)
0619 01:08:44.096652 (+ 22126us) negotiation.cc:107] Waiting for socket to connect
0619 01:08:44.096738 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0619 01:08:44.100542 (+ 3804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:08:44.102512 (+ 1970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:08:44.102523 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:08:44.103019 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:08:44.103649 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:44.103669 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:44.105028 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:44.105032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:44.106471 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:44.106485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:44.107849 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:44.107857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:44.108198 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:44.109147 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:08:44.109183 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:08:44.112260 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:08:44.117269 (+ 5009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:08:44.117280 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:08:44.117323 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:08:44.118043 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:08:44.118631 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:08:44.118637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:08:44.118641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:08:44.118937 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:08:44.119653 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:08:44.119663 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:08:44.120195 (+ 532us) client_negotiation.cc:769] Sending connection context
0619 01:08:44.120599 (+ 404us) client_negotiation.cc:240] Negotiation successful
0619 01:08:44.122325 (+ 1726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21496,"thread_start_us":21164,"threads_started":1}
I20260619 01:08:44.126403 11926 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37277
I20260619 01:08:44.126796 11926 heartbeater.cc:464] Registering TS with master...
I20260619 01:08:44.127782 11926 heartbeater.cc:511] Master 127.0.97.62:37277 requested a full tablet report, sending...
I20260619 01:08:44.130604 11713 ts_manager.cc:194] Registered new tserver with Master: 6c9d3f0ad0c44b0f892736eeb1859a13 (127.0.97.1:42587)
I20260619 01:08:44.132740 11713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:08:44.161878 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:08:44 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 01:08:44.674923 11932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:08:44.675235 11932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:08:44.675414 11932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:08:44.683022 11932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:08:44.683151 11932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:08:44.683332 11932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:08:44.693970 11932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:49921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:08:44.695360 11932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:08:44.696798 11932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:08:44.711879 11946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:08:44.711959 11947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:08:44.713049 11949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:08:44.713573 11932 server_base.cc:1034] running on GCE node
I20260619 01:08:44.714277 11932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:08:44.715360 11932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:08:44.716630 11932 hybrid_clock.cc:648] HybridClock initialized: now 1781831324716570 us; error 78 us; skew 500 ppm
I20260619 01:08:44.717068 11932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:08:44 dist-test-slave-5p8w krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:08:44.720743 11932 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:08:44.722503 11932 webserver.cc:460] Webserver started at http://127.0.97.2:36767/ using document root <none> and password file <none>
I20260619 01:08:44.723330 11932 fs_manager.cc:359] Metadata directory not provided
I20260619 01:08:44.723455 11932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:08:44.723729 11932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:08:44 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:08:44.742216 11932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "1f67e87c1ee44a98a3f2cc97a262b27c"
format_stamp: "Formatted at 2026-06-19 01:08:44 on dist-test-slave-5p8w"
server_key: "7f59af2c645f711f0c38591b35c6dd33"
server_key_iv: "eef9d094140ba0ecefdabffc04c88b35"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:44.743443 11932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1f67e87c1ee44a98a3f2cc97a262b27c"
format_stamp: "Formatted at 2026-06-19 01:08:44 on dist-test-slave-5p8w"
server_key: "7f59af2c645f711f0c38591b35c6dd33"
server_key_iv: "eef9d094140ba0ecefdabffc04c88b35"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:44.749661 11932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 01:08:44.773123 11956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:08:44.775224 11932 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.001s
I20260619 01:08:44.775480 11932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "1f67e87c1ee44a98a3f2cc97a262b27c"
format_stamp: "Formatted at 2026-06-19 01:08:44 on dist-test-slave-5p8w"
server_key: "7f59af2c645f711f0c38591b35c6dd33"
server_key_iv: "eef9d094140ba0ecefdabffc04c88b35"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:44.775740 11932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:08:44.802947 11932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:08:44.807345 11932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:08:44.807677 11932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:08:44.808949 11932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:08:44.811095 11932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:08:44.811208 11932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:08:44.811347 11932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:08:44.811417 11932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:08:44.858981 11932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36621
I20260619 01:08:44.858999 12068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36621 every 8 connection(s)
I20260619 01:08:44.860510 11932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:08:44.870589 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 11932
I20260619 01:08:44.870826 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:08:44.871546 388 external_mini_cluster.cc:1428] Setting key 557385064e755b35261273311fecf719
Jun 19 01:08:44 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:08:44.904953 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:44.863937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55009 (local address 127.0.97.62:37277)
0619 01:08:44.881982 (+ 18045us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:44.881990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:44.886304 (+ 4314us) server_negotiation.cc:407] Connection header received
0619 01:08:44.886381 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:44.886386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:44.886479 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:44.886617 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:08:44.888877 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:44.890100 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:44.892767 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:44.893407 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:44.893527 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:44.897967 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:08:44.897973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:08:44.897976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:08:44.898031 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:08:44.900831 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:44.901732 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:44.901739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:44.901744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:44.901887 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:44.902362 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:44.902367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:44.902375 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:44.902651 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:08:44.902799 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:44.903535 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 01:08:44.903770 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17877,"thread_start_us":18061,"threads_started":1}
I20260619 01:08:44.905535 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:44.864868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37277 (local address 127.0.97.2:55009)
0619 01:08:44.882723 (+ 17855us) negotiation.cc:107] Waiting for socket to connect
0619 01:08:44.882798 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 01:08:44.885849 (+ 3051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:08:44.887065 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:08:44.887077 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:08:44.887740 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:08:44.888647 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:44.888677 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:44.890397 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:44.890403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:44.892556 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:44.892578 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:44.893892 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:44.893899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:44.894216 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:44.895080 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:08:44.895112 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:08:44.897672 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:08:44.901000 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:08:44.901010 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:08:44.901030 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:08:44.901596 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:08:44.902001 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:08:44.902006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:08:44.902009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:08:44.902235 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:08:44.902779 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:08:44.902789 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:08:44.903151 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 01:08:44.903424 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 01:08:44.904004 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17492,"thread_start_us":17235,"threads_started":1}
I20260619 01:08:44.907287 12069 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37277
I20260619 01:08:44.907640 12069 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 01:08:44.908349 12069 heartbeater.cc:511] Master 127.0.97.62:37277 requested a full tablet report, sending...
I20260619 01:08:44.909891 11713 ts_manager.cc:194] Registered new tserver with Master: 1f67e87c1ee44a98a3f2cc97a262b27c (127.0.97.2:36621)
I20260619 01:08:44.910739 11713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:08:44.942799 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:49921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:08:45.137192 11926 heartbeater.cc:503] Master 127.0.97.62:37277 was elected leader, sending a full tablet report...
W20260619 01:08:45.342955 12075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:08:45.343354 12075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:08:45.343523 12075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:08:45.350951 12075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:08:45.351105 12075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:08:45.351276 12075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:08:45.361400 12075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:49921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:08:45.362780 12075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:08:45.364252 12075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:08:45.378494 12083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:08:45.378541 12081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:08:45.378559 12080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:08:45.379535 12075 server_base.cc:1034] running on GCE node
I20260619 01:08:45.380308 12075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:08:45.381413 12075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:08:45.382685 12075 hybrid_clock.cc:648] HybridClock initialized: now 1781831325382611 us; error 68 us; skew 500 ppm
I20260619 01:08:45.383059 12075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:08:45 dist-test-slave-5p8w krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:08:45.386250 12075 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:08:45.387816 12075 webserver.cc:460] Webserver started at http://127.0.97.3:40975/ using document root <none> and password file <none>
I20260619 01:08:45.388512 12075 fs_manager.cc:359] Metadata directory not provided
I20260619 01:08:45.388615 12075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:08:45.388873 12075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:08:45 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:08:45.403971 12075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "2a3f70bc87ec4b418218fc74f7dbdf75"
format_stamp: "Formatted at 2026-06-19 01:08:45 on dist-test-slave-5p8w"
server_key: "8198d75cef77ea0ab962d9a5f1768c4e"
server_key_iv: "a0ac5b727aaf4c58c42a029fd37fa7a6"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:45.404801 12075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2a3f70bc87ec4b418218fc74f7dbdf75"
format_stamp: "Formatted at 2026-06-19 01:08:45 on dist-test-slave-5p8w"
server_key: "8198d75cef77ea0ab962d9a5f1768c4e"
server_key_iv: "a0ac5b727aaf4c58c42a029fd37fa7a6"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:45.410766 12075 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 01:08:45.426623 12090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:08:45.428107 12075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 01:08:45.428308 12075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "2a3f70bc87ec4b418218fc74f7dbdf75"
format_stamp: "Formatted at 2026-06-19 01:08:45 on dist-test-slave-5p8w"
server_key: "8198d75cef77ea0ab962d9a5f1768c4e"
server_key_iv: "a0ac5b727aaf4c58c42a029fd37fa7a6"
server_key_version: "kuduclusterkey@0"
I20260619 01:08:45.428561 12075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:08:45.443639 12075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:08:45.447551 12075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:08:45.447857 12075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:08:45.448797 12075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:08:45.450716 12075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:08:45.450824 12075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260619 01:08:45.450984 12075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:08:45.451056 12075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:08:45.487097 12075 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39615
I20260619 01:08:45.487133 12202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39615 every 8 connection(s)
I20260619 01:08:45.488444 12075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:08:45.490542 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 12075
I20260619 01:08:45.490734 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:08:45.491221 388 external_mini_cluster.cc:1428] Setting key abb2fd76c55dc0209348f38fdb5ca664
Jun 19 01:08:45 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:08:45.509626 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.491067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52813 (local address 127.0.97.62:37277)
0619 01:08:45.491658 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:45.491665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:45.492628 (+ 963us) server_negotiation.cc:407] Connection header received
0619 01:08:45.494487 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:45.494491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:45.494575 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:45.494664 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:08:45.496083 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.496779 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.498397 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.498847 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.498950 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.502964 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:08:45.502969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:08:45.502973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:08:45.503007 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:08:45.506204 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:45.507131 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:45.507136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:45.507140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:45.507305 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:45.507764 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:45.507774 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:45.507779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:45.508139 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:08:45.508263 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:45.509084 (+ 821us) server_negotiation.cc:299] Negotiation successful
0619 01:08:45.509265 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":251,"threads_started":1}
I20260619 01:08:45.510623 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.491669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37277 (local address 127.0.97.3:52813)
0619 01:08:45.492470 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 01:08:45.492526 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 01:08:45.494258 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:08:45.494881 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:08:45.494889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:08:45.495339 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:08:45.495920 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.495936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.496912 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.496915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.498264 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.498277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.499489 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.499496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.499714 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.500489 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:08:45.500513 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:08:45.502709 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:08:45.506378 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:08:45.506388 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:08:45.506417 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:08:45.506964 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:08:45.507408 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:08:45.507412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:08:45.507414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:08:45.507635 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:08:45.508255 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:08:45.508262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:08:45.508698 (+ 436us) client_negotiation.cc:769] Sending connection context
0619 01:08:45.508993 (+ 295us) client_negotiation.cc:240] Negotiation successful
0619 01:08:45.509478 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":349,"threads_started":1}
I20260619 01:08:45.512202 12203 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37277
I20260619 01:08:45.512490 12203 heartbeater.cc:464] Registering TS with master...
I20260619 01:08:45.513202 12203 heartbeater.cc:511] Master 127.0.97.62:37277 requested a full tablet report, sending...
I20260619 01:08:45.514722 11713 ts_manager.cc:194] Registered new tserver with Master: 2a3f70bc87ec4b418218fc74f7dbdf75 (127.0.97.3:39615)
I20260619 01:08:45.515630 11713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52813
I20260619 01:08:45.519693 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:08:45.537894 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.524474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41328 (local address 127.0.97.62:37277)
0619 01:08:45.524697 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:45.524706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:45.525587 (+ 881us) server_negotiation.cc:407] Connection header received
0619 01:08:45.525794 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:45.525800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:45.525889 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:45.525997 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:08:45.527433 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.528168 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.529629 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.530069 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.530207 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.532781 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:08:45.532786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:08:45.532790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:08:45.532829 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:08:45.535359 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:45.536130 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:45.536134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:45.536136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:45.536252 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:45.536579 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:45.536582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:45.536584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:45.536851 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:08:45.537031 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:45.537419 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 01:08:45.537600 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:08:45.541970 11713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:08:45.549206 11713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:08:45.574898 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.563229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.0.97.3:39615)
0619 01:08:45.563967 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:45.563975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:45.564012 (+ 37us) server_negotiation.cc:407] Connection header received
0619 01:08:45.564212 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:45.564218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:45.564496 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:45.564710 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:08:45.566063 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.568026 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.571378 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.572901 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.573038 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.573345 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:45.574334 (+ 989us) server_negotiation.cc:299] Negotiation successful
0619 01:08:45.574509 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":384,"threads_started":1}
I20260619 01:08:45.575049 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.564222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36621 (local address 127.0.0.1:57146)
0619 01:08:45.564814 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 01:08:45.564832 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:08:45.564943 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:08:45.566809 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:08:45.566814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:08:45.566838 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:08:45.567179 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.567187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.569331 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.569336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.571136 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.571149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.573794 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.573806 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.574222 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.574254 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:08:45.574362 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 01:08:45.574512 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":53,"thread_start_us":321,"threads_started":1}
I20260619 01:08:45.575431 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.563002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39615 (local address 127.0.0.1:37730)
0619 01:08:45.563740 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 01:08:45.563786 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:08:45.563973 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:08:45.565115 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:08:45.565127 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:08:45.565181 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:08:45.565832 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.565848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.568160 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.568165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.571170 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.571187 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.573903 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.573909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.574073 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.574144 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 01:08:45.574282 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 01:08:45.574496 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":2,"thread_start_us":326,"threads_started":1}
I20260619 01:08:45.575706 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.564873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42587 (local address 127.0.0.1:55216)
0619 01:08:45.565527 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 01:08:45.565550 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:08:45.565668 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:08:45.567494 (+ 1826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:08:45.567499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:08:45.567521 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:08:45.567886 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.567897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.569491 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.569495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.571688 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.571705 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.573878 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.573887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.574565 (+ 678us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.574594 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 01:08:45.574709 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 01:08:45.574840 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":456,"thread_start_us":857,"threads_started":1}
I20260619 01:08:45.575716 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.564473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.0.97.2:36621)
0619 01:08:45.566164 (+ 1691us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:45.566173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:45.566200 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:08:45.566307 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:45.566314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:45.566582 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:45.567350 (+ 768us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:08:45.567603 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.569188 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.571307 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.573086 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.574018 (+ 932us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.574221 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:45.575153 (+ 932us) server_negotiation.cc:299] Negotiation successful
0619 01:08:45.575332 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1533,"thread_start_us":1408,"threads_started":1}
I20260619 01:08:45.575791 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.565524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.0.97.1:42587)
0619 01:08:45.566933 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:45.566941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:45.566972 (+ 31us) server_negotiation.cc:407] Connection header received
0619 01:08:45.567076 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:45.567083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:45.567291 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:45.567474 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:08:45.568060 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.569347 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.571900 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.573404 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.574161 (+ 757us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.574400 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:45.575264 (+ 864us) server_negotiation.cc:299] Negotiation successful
0619 01:08:45.575453 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":1145,"threads_started":1}
I20260619 01:08:45.579839 12004 tablet_service.cc:1467] Processing CreateTablet for tablet 0b96016a72f94f91b3a04be9a38acb5d (DEFAULT_TABLE table=test-table [id=ade58ce6b6a74fbbab1271e9dc3c9d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:08:45.579845 12138 tablet_service.cc:1467] Processing CreateTablet for tablet 0b96016a72f94f91b3a04be9a38acb5d (DEFAULT_TABLE table=test-table [id=ade58ce6b6a74fbbab1271e9dc3c9d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:08:45.580462 11861 tablet_service.cc:1467] Processing CreateTablet for tablet 0b96016a72f94f91b3a04be9a38acb5d (DEFAULT_TABLE table=test-table [id=ade58ce6b6a74fbbab1271e9dc3c9d88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:08:45.581470 12138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b96016a72f94f91b3a04be9a38acb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:08:45.581470 12004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b96016a72f94f91b3a04be9a38acb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:08:45.581908 11861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b96016a72f94f91b3a04be9a38acb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:08:45.594563 12222 tablet_bootstrap.cc:492] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13: Bootstrap starting.
I20260619 01:08:45.597640 12224 tablet_bootstrap.cc:492] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: Bootstrap starting.
I20260619 01:08:45.598975 12223 tablet_bootstrap.cc:492] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: Bootstrap starting.
I20260619 01:08:45.601281 12222 tablet_bootstrap.cc:654] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13: Neither blocks nor log segments found. Creating new log.
I20260619 01:08:45.601281 12224 tablet_bootstrap.cc:654] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: Neither blocks nor log segments found. Creating new log.
I20260619 01:08:45.602277 12223 tablet_bootstrap.cc:654] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: Neither blocks nor log segments found. Creating new log.
I20260619 01:08:45.603123 12222 log.cc:824] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13: Log is configured to *not* fsync() on all Append() calls
I20260619 01:08:45.603123 12224 log.cc:824] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: Log is configured to *not* fsync() on all Append() calls
I20260619 01:08:45.603592 12223 log.cc:824] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: Log is configured to *not* fsync() on all Append() calls
I20260619 01:08:45.606950 12223 tablet_bootstrap.cc:492] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: No bootstrap required, opened a new log
I20260619 01:08:45.606976 12222 tablet_bootstrap.cc:492] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13: No bootstrap required, opened a new log
I20260619 01:08:45.607252 12223 ts_tablet_manager.cc:1397] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 01:08:45.607265 12222 ts_tablet_manager.cc:1397] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260619 01:08:45.607350 12224 tablet_bootstrap.cc:492] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: No bootstrap required, opened a new log
I20260619 01:08:45.607613 12224 ts_tablet_manager.cc:1397] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 01:08:45.615057 12223 raft_consensus.cc:348] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39615 } } peers { permanent_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36621 } } peers { permanent_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42587 } }
I20260619 01:08:45.615526 12223 raft_consensus.cc:374] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:08:45.615631 12223 raft_consensus.cc:729] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f67e87c1ee44a98a3f2cc97a262b27c, State: Initialized, Role: FOLLOWER
I20260619 01:08:45.616374 12223 consensus_queue.cc:260] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39615 } } peers { permanent_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36621 } } peers { permanent_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42587 } }
I20260619 01:08:45.619179 12069 heartbeater.cc:503] Master 127.0.97.62:37277 was elected leader, sending a full tablet report...
I20260619 01:08:45.619158 12222 raft_consensus.cc:348] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39615 } } peers { permanent_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36621 } } peers { permanent_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42587 } }
I20260619 01:08:45.619704 12222 raft_consensus.cc:374] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:08:45.619871 12222 raft_consensus.cc:729] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9d3f0ad0c44b0f892736eeb1859a13, State: Initialized, Role: FOLLOWER
I20260619 01:08:45.620009 12223 ts_tablet_manager.cc:1428] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260619 01:08:45.620179 12224 raft_consensus.cc:348] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39615 } } peers { permanent_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36621 } } peers { permanent_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42587 } }
I20260619 01:08:45.620692 12224 raft_consensus.cc:374] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:08:45.620808 12224 raft_consensus.cc:729] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3f70bc87ec4b418218fc74f7dbdf75, State: Initialized, Role: FOLLOWER
I20260619 01:08:45.620739 12222 consensus_queue.cc:260] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39615 } } peers { permanent_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36621 } } peers { permanent_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42587 } }
I20260619 01:08:45.621560 12224 consensus_queue.cc:260] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39615 } } peers { permanent_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36621 } } peers { permanent_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42587 } }
I20260619 01:08:45.622821 12222 ts_tablet_manager.cc:1428] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260619 01:08:45.623885 12203 heartbeater.cc:503] Master 127.0.97.62:37277 was elected leader, sending a full tablet report...
I20260619 01:08:45.624755 12224 ts_tablet_manager.cc:1428] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
W20260619 01:08:45.638090 12070 tablet.cc:2369] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:08:45.686975 12228 raft_consensus.cc:482] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:08:45.687347 12228 raft_consensus.cc:504] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39615 } } peers { permanent_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36621 } } peers { permanent_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42587 } }
I20260619 01:08:45.689111 12228 leader_election.cc:288] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a3f70bc87ec4b418218fc74f7dbdf75 (127.0.97.3:39615), 6c9d3f0ad0c44b0f892736eeb1859a13 (127.0.97.1:42587)
I20260619 01:08:45.698344 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.689724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59695 (local address 127.0.97.3:39615)
0619 01:08:45.689972 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:45.689980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:45.690417 (+ 437us) server_negotiation.cc:407] Connection header received
0619 01:08:45.690673 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:45.690679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:45.690757 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:45.690823 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:08:45.692098 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.693134 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.695761 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.697056 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.697204 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.697494 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:45.697948 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 01:08:45.698088 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:08:45.698347 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.690503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36789 (local address 127.0.97.1:42587)
0619 01:08:45.690719 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:45.690725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:45.691250 (+ 525us) server_negotiation.cc:407] Connection header received
0619 01:08:45.691447 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:45.691454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:45.691552 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:45.691641 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:08:45.692477 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.693502 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.695771 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.697059 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.697204 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.697349 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:45.697949 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 01:08:45.698102 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 01:08:45.698628 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.690355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42587 (local address 127.0.97.2:36789)
0619 01:08:45.691132 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 01:08:45.691148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:08:45.691300 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:08:45.691970 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:08:45.691974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:08:45.692002 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:08:45.692326 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.692333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.693631 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.693635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.695626 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.695639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.697404 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.697409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.697712 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.697793 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 01:08:45.697897 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 01:08:45.698107 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":101,"thread_start_us":496,"threads_started":1}
I20260619 01:08:45.698839 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.689486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39615 (local address 127.0.97.2:59695)
0619 01:08:45.690296 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 01:08:45.690318 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:08:45.690484 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:08:45.691033 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:08:45.691053 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:08:45.691080 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:08:45.691946 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.691960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.693272 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.693275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.695542 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:08:45.695560 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.697392 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.697398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:08:45.697705 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.697795 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 01:08:45.697897 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 01:08:45.698107 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":149,"thread_start_us":421,"threads_started":1}
I20260619 01:08:45.699805 12158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b96016a72f94f91b3a04be9a38acb5d" candidate_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" is_pre_election: true
I20260619 01:08:45.699779 11881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b96016a72f94f91b3a04be9a38acb5d" candidate_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" is_pre_election: true
I20260619 01:08:45.700305 12158 raft_consensus.cc:2454] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f67e87c1ee44a98a3f2cc97a262b27c in term 0.
I20260619 01:08:45.700304 11881 raft_consensus.cc:2454] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f67e87c1ee44a98a3f2cc97a262b27c in term 0.
I20260619 01:08:45.701009 11958 leader_election.cc:302] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f67e87c1ee44a98a3f2cc97a262b27c, 6c9d3f0ad0c44b0f892736eeb1859a13; no voters:
I20260619 01:08:45.701442 12228 raft_consensus.cc:2781] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:08:45.701591 12228 raft_consensus.cc:482] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:08:45.701730 12228 raft_consensus.cc:3037] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:08:45.704021 12228 raft_consensus.cc:504] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39615 } } peers { permanent_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36621 } } peers { permanent_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42587 } }
I20260619 01:08:45.704738 12228 leader_election.cc:288] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [CANDIDATE]: Term 1 election: Requested vote from peers 2a3f70bc87ec4b418218fc74f7dbdf75 (127.0.97.3:39615), 6c9d3f0ad0c44b0f892736eeb1859a13 (127.0.97.1:42587)
I20260619 01:08:45.705401 12158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b96016a72f94f91b3a04be9a38acb5d" candidate_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75"
I20260619 01:08:45.705523 11881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b96016a72f94f91b3a04be9a38acb5d" candidate_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13"
I20260619 01:08:45.705641 12158 raft_consensus.cc:3037] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:08:45.705783 11881 raft_consensus.cc:3037] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:08:45.708352 12158 raft_consensus.cc:2454] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f67e87c1ee44a98a3f2cc97a262b27c in term 1.
I20260619 01:08:45.708544 11881 raft_consensus.cc:2454] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f67e87c1ee44a98a3f2cc97a262b27c in term 1.
I20260619 01:08:45.708968 11958 leader_election.cc:302] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f67e87c1ee44a98a3f2cc97a262b27c, 2a3f70bc87ec4b418218fc74f7dbdf75; no voters:
I20260619 01:08:45.709326 12228 raft_consensus.cc:2781] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:08:45.709822 12228 raft_consensus.cc:686] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 1 LEADER]: Becoming Leader. State: Replica: 1f67e87c1ee44a98a3f2cc97a262b27c, State: Running, Role: LEADER
I20260619 01:08:45.710328 12228 consensus_queue.cc:237] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39615 } } peers { permanent_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36621 } } peers { permanent_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42587 } }
I20260619 01:08:45.715224 11712 catalog_manager.cc:5495] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f67e87c1ee44a98a3f2cc97a262b27c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f67e87c1ee44a98a3f2cc97a262b27c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20260619 01:08:45.734246 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.728338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.0.97.2:36621)
0619 01:08:45.728537 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:45.728541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:45.728558 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:08:45.728732 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:45.728736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:45.728822 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:45.728951 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:08:45.729330 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.730058 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.731973 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.732382 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.732517 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.732988 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:08:45.733637 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:08:45.733755 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:45.733881 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 01:08:45.733945 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 01:08:45.740067 12204 tablet.cc:2369] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:08:45.742620 12158 raft_consensus.cc:1261] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75 [term 1 FOLLOWER]: Refusing update from remote peer 1f67e87c1ee44a98a3f2cc97a262b27c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:08:45.742619 11881 raft_consensus.cc:1261] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13 [term 1 FOLLOWER]: Refusing update from remote peer 1f67e87c1ee44a98a3f2cc97a262b27c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:08:45.743646 12228 consensus_queue.cc:1035] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a3f70bc87ec4b418218fc74f7dbdf75" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:08:45.744010 12233 consensus_queue.cc:1035] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9d3f0ad0c44b0f892736eeb1859a13" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:08:45.753880 12236 mvcc.cc:205] Tried to move back new op lower bound from 7298381110231150592 to 7298381110113345536. Current Snapshot: MvccSnapshot[applied={T|T < 7298381110231150592}]
I20260619 01:08:45.759487 12238 mvcc.cc:205] Tried to move back new op lower bound from 7298381110231150592 to 7298381110113345536. Current Snapshot: MvccSnapshot[applied={T|T < 7298381110231150592}]
I20260619 01:08:45.764631 12237 mvcc.cc:205] Tried to move back new op lower bound from 7298381110231150592 to 7298381110113345536. Current Snapshot: MvccSnapshot[applied={T|T < 7298381110231150592}]
I20260619 01:08:45.776485 11712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:08:45.776798 11712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:08:45.780629 11712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55fa0202bc2d422e9e27ea5dea98c177: Sending DeleteTablet for 3 replicas of tablet 0b96016a72f94f91b3a04be9a38acb5d
I20260619 01:08:45.781891 12138 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b96016a72f94f91b3a04be9a38acb5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:08:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37730
I20260619 01:08:45.782213 11861 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b96016a72f94f91b3a04be9a38acb5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:08:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55216
I20260619 01:08:45.782222 12004 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b96016a72f94f91b3a04be9a38acb5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:08:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57146
I20260619 01:08:45.785354 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 11793
I20260619 01:08:45.788129 12248 tablet_replica.cc:330] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13: stopping tablet replica
I20260619 01:08:45.797149 12246 tablet_replica.cc:330] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: stopping tablet replica
I20260619 01:08:45.797698 12246 raft_consensus.cc:2229] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:08:45.797703 12248 raft_consensus.cc:2229] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:08:45.798254 12248 raft_consensus.cc:2258] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:08:45.798247 12246 raft_consensus.cc:2258] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:08:45.799144 12247 tablet_replica.cc:330] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: stopping tablet replica
I20260619 01:08:45.799602 12247 raft_consensus.cc:2229] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:08:45.800287 12247 raft_consensus.cc:2258] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:08:45.800536 12246 ts_tablet_manager.cc:1905] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:08:45.802235 12247 ts_tablet_manager.cc:1905] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:08:45.802295 12248 ts_tablet_manager.cc:1905] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:08:45.808353 12246 ts_tablet_manager.cc:1918] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:08:45.808528 12246 log.cc:1196] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/0b96016a72f94f91b3a04be9a38acb5d
I20260619 01:08:45.809006 12246 ts_tablet_manager.cc:1939] T 0b96016a72f94f91b3a04be9a38acb5d P 2a3f70bc87ec4b418218fc74f7dbdf75: Deleting consensus metadata
I20260619 01:08:45.809537 12247 ts_tablet_manager.cc:1918] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:08:45.809723 12247 log.cc:1196] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/0b96016a72f94f91b3a04be9a38acb5d
I20260619 01:08:45.810199 12247 ts_tablet_manager.cc:1939] T 0b96016a72f94f91b3a04be9a38acb5d P 1f67e87c1ee44a98a3f2cc97a262b27c: Deleting consensus metadata
I20260619 01:08:45.810187 12248 ts_tablet_manager.cc:1918] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 19 01:08:45 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:08:45.810442 12248 log.cc:1196] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/0b96016a72f94f91b3a04be9a38acb5d
I20260619 01:08:45.810951 12248 ts_tablet_manager.cc:1939] T 0b96016a72f94f91b3a04be9a38acb5d P 6c9d3f0ad0c44b0f892736eeb1859a13: Deleting consensus metadata
I20260619 01:08:45.811082 11698 catalog_manager.cc:4841] TS 2a3f70bc87ec4b418218fc74f7dbdf75 (127.0.97.3:39615): tablet 0b96016a72f94f91b3a04be9a38acb5d (table test-table [id=ade58ce6b6a74fbbab1271e9dc3c9d88]) successfully deleted
I20260619 01:08:45.811816 11698 catalog_manager.cc:4841] TS 1f67e87c1ee44a98a3f2cc97a262b27c (127.0.97.2:36621): tablet 0b96016a72f94f91b3a04be9a38acb5d (table test-table [id=ade58ce6b6a74fbbab1271e9dc3c9d88]) successfully deleted
I20260619 01:08:45.812736 11698 catalog_manager.cc:4841] TS 6c9d3f0ad0c44b0f892736eeb1859a13 (127.0.97.1:42587): tablet 0b96016a72f94f91b3a04be9a38acb5d (table test-table [id=ade58ce6b6a74fbbab1271e9dc3c9d88]) successfully deleted
I20260619 01:08:45.818636 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.787235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.0.97.1:42587)
0619 01:08:45.787516 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:45.787526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:45.799503 (+ 11977us) server_negotiation.cc:407] Connection header received
0619 01:08:45.799749 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:45.799754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:45.799858 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:45.799957 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:08:45.802020 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.803083 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.805158 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.805809 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.805928 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.811962 (+ 6034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:08:45.811976 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:08:45.811994 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:08:45.812044 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:08:45.815391 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:45.816160 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:45.816168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:45.816171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:45.816307 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:45.816797 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:45.816804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:45.816807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:45.817179 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:08:45.817564 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:45.817965 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 01:08:45.818194 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:08:45.819381 11921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:08:45.930886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 11793
I20260619 01:08:45.949120 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 11932
Jun 19 01:08:45 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:08:45.967337 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:45.950041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.0.97.2:36621)
0619 01:08:45.950232 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:45.950239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:45.950261 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:08:45.950406 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:45.950412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:45.950492 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:45.950614 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:08:45.952024 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.953440 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.954953 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:45.955661 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:45.955799 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:45.960595 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:08:45.960609 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:08:45.960625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:08:45.960665 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:08:45.963996 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:45.965078 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:45.965086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:45.965089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:45.965248 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:45.965795 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:45.965799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:45.965801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:45.966213 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:08:45.966464 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:45.966812 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 01:08:45.966991 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:08:45.968106 12064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:08:46.084936 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 11932
I20260619 01:08:46.103835 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 12075
Jun 19 01:08:46 dist-test-slave-5p8w krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:08:46.119470 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:08:46.104600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.0.97.3:39615)
0619 01:08:46.104780 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 01:08:46.104788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:08:46.104879 (+ 91us) server_negotiation.cc:407] Connection header received
0619 01:08:46.105093 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:08:46.105105 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:08:46.105167 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:08:46.105284 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:08:46.106591 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:46.107325 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:46.108637 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:08:46.109078 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:08:46.109213 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:08:46.113641 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:08:46.113662 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:08:46.113678 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:08:46.113714 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:08:46.116710 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:46.117591 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:46.117599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:46.117601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:46.117776 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:08:46.118155 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:08:46.118158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:08:46.118160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:08:46.118597 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:08:46.118809 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 01:08:46.119061 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 01:08:46.119220 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:08:46.120141 12198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:08:46.211920 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 12075
I20260619 01:08:46.228830 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 11681
I20260619 01:08:46.230074 11743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:08:46.343065 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 11681
I20260619 01:08:46.361027 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 01:08:46.696799 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 01:08:46.697003 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 01:08:47.042798 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 01:08:47.043 UTC [11267] LOG: received smart shutdown request
2026-06-19 01:08:47.046 UTC [11267] LOG: background worker "logical replication launcher" (PID 11276) exited with exit code 1
2026-06-19 01:08:47.047 UTC [11271] LOG: shutting down
2026-06-19 01:08:47.228 UTC [11267] LOG: database system is shut down
2026-06-19T01:08:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52995 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:08:47 dist-test-slave-5p8w krb5kdc[12262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:08:47 dist-test-slave-5p8w krb5kdc[12262](info): set up 2 sockets
Jun 19 01:08:47 dist-test-slave-5p8w krb5kdc[12262](info): commencing operation
krb5kdc: starting...
W20260619 01:08:48.481655 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:08:48 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:08:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:08:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 01:08:48.818782 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbdbgje/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/postgres -l logfile start
2026-06-19 01:08:51.343 UTC [12292] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 01:08:51.343 UTC [12292] LOG: listening on IPv4 address "127.0.97.20", port 35509
2026-06-19 01:08:51.346 UTC [12292] LOG: listening on Unix socket "/tmp/.s.PGSQL.35509"
2026-06-19 01:08:51.367 UTC [12295] LOG: database system was shut down at 2026-06-19 01:08:49 UTC
2026-06-19 01:08:51.371 UTC [12292] LOG: database system is ready to accept connections
I20260619 01:08:52.345266 388 mini_postgres.cc:96] Postgres bound to 35509
2026-06-19 01:08:52.362 UTC [12303] FATAL: database "slave" does not exist
127.0.97.20:35509 - accepting connections
I20260619 01:08:52.363253 388 mini_ranger.cc:161] Starting Ranger...
I20260619 01:08:52.389891 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 01:08:52.873134 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 01:08:52.873265 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-admin
2026-06-19 01:08:53,161 [I] DB FLAVOR :POSTGRES
2026-06-19 01:08:53,161 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 01:08:53,161 [I] Checking connection..
2026-06-19 01:08:53,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 01:08:53,425 [I] Checking connection passed.
2026-06-19 01:08:53,425 [I] --------- Verifying version history table ---------
2026-06-19 01:08:53,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:08:53,669 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 01:08:53,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:08:53,928 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 01:08:53,928 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 01:08:53,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 01:08:54.208 UTC [12367] WARNING: there is no transaction in progress
2026-06-19 01:08:54,214 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 01:08:54,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:08:54,464 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 01:08:54,464 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 01:08:54,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 01:08:54,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 01:08:54,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5p8w.c.gcp-upstream.internal','N') ;"
2026-06-19 01:08:55,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 01:08:55,478 [I] Table x_portal_user does not exist in database ranger
2026-06-19 01:08:55,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 01:08:55,726 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 01:08:55,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 01:08:55,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 01:08:56,255 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 01:08:56,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 01:08:57.238 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.257 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.274 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.290 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.310 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.327 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.386 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.394 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.410 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.421 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.432 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.447 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.457 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.464 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.471 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:57.478 UTC [12475] WARNING: there is no transaction in progress
2026-06-19 01:08:58,147 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 01:08:58,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 01:08:58,380 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 01:08:58,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 01:08:58,635 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 01:08:58,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 01:08:58,918 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 01:08:58,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 01:08:59,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 01:08:59,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5p8w.c.gcp-upstream.internal';"
2026-06-19 01:08:59,699 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 01:08:59,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 01:08:59,971 [I] DB_PATCHES have already been applied
I20260619 01:08:59.978055 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbdbgje/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 01:08:59.978142 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 01:08:59.993054 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:44077
Jun 19, 2026 1:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 1:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 1:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 1:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 1:09:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 01:09:01.158432 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.795s user 0.005s sys 0.035s
Jun 19, 2026 1:09:22 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 1:09:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 1:09:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 1:09:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 1:09:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 01:09:23.858559 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 01:09:24 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831364, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 01:09:24.052088 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 01:09:24.073668 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 01:09:25.045033 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 01:09:25.045133 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-kms
I20260619 01:09:25.045171 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:35509
2026-06-19 01:09:25,404 [I] DB FLAVOR :POSTGRES
2026-06-19 01:09:25,404 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 01:09:25,404 [I] Checking connection
2026-06-19 01:09:25,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 01:09:25,656 [I] connection success
2026-06-19 01:09:25,656 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 01:09:25,656 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 01:09:25,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 01:09:25,914 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 01:09:25,914 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 01:09:25,914 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 01:09:25,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35509/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 01:09:26,214 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 01:09:26.694278 388 mini_ranger_kms.cc:310] Created kms service
I20260619 01:09:26.952266 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 01:09:27.007617 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 01:09:27 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831364, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:09:27.459699 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 01:09:27.459995 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbdbgje/build/asan/bin/postgresql.jar:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbdbgje/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 01:09:27.460041 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 01:09:27.478168 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12656
I20260619 01:09:27.478307 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:40457
Jun 19, 2026 1:09:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:09:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:09:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 1:09:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 1:09:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 1:09:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 01:09:29.716312 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.664s user 0.059s sys 0.309s
I20260619 01:09:29.716586 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 01:09:29.716771 388 mini_ranger_kms.cc:401] 127.0.97.20:40457/kms/v1/keys
Jun 19 01:09:29 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781831369, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 01:09:30 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831369, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 01:09:30 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831369, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 01:09:32 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:09:33.038790 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34645
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34645
--ranger_config_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40457/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:09:33.379426 12706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:33.379786 12706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:33.379938 12706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:33.387372 12706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:09:33.387531 12706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:33.387636 12706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:33.387739 12706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:09:33.387820 12706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:09:33.398370 12706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:40457/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34645
--ranger_config_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:33.399711 12706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:33.401041 12706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:33.414806 12711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:33.414826 12712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:33.418417 12714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:33.419266 12706 server_base.cc:1034] running on GCE node
I20260619 01:09:33.420116 12706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:33.421723 12706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:33.422987 12706 hybrid_clock.cc:648] HybridClock initialized: now 1781831373422939 us; error 37 us; skew 500 ppm
I20260619 01:09:33.423393 12706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:33 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:33.426914 12706 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:09:33.428644 12706 webserver.cc:460] Webserver started at http://127.0.97.62:39567/ using document root <none> and password file <none>
I20260619 01:09:33.429462 12706 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:33.429601 12706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:33.429915 12706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:09:33 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:09:33.489516 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "01bea5ad3bca4eb49d036006597d8d0d"
format_stamp: "Formatted at 2026-06-19 01:09:33 on dist-test-slave-5p8w"
server_key: "4b7f659455b4830af34a31fd29b7887b"
server_key_iv: "5c833c4c97d22aec385d54fd187fb2ca"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:33.490623 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "01bea5ad3bca4eb49d036006597d8d0d"
format_stamp: "Formatted at 2026-06-19 01:09:33 on dist-test-slave-5p8w"
server_key: "4b7f659455b4830af34a31fd29b7887b"
server_key_iv: "5c833c4c97d22aec385d54fd187fb2ca"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:33.497896 12706 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260619 01:09:33.515481 12721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:33.517120 12706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 01:09:33.517335 12706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "01bea5ad3bca4eb49d036006597d8d0d"
format_stamp: "Formatted at 2026-06-19 01:09:33 on dist-test-slave-5p8w"
server_key: "4b7f659455b4830af34a31fd29b7887b"
server_key_iv: "5c833c4c97d22aec385d54fd187fb2ca"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:33.517555 12706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:33.554350 12706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:33.558504 12706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:33.558851 12706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:33.577216 12706 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34645
I20260619 01:09:33.577251 12772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34645 every 8 connection(s)
I20260619 01:09:33.578650 12706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:09:33.582562 12773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:33.586058 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 12706
I20260619 01:09:33.586310 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:09:33.587121 388 external_mini_cluster.cc:1428] Setting key 61554fbe7f9ea920d9601bd7039da251
I20260619 01:09:33.595594 12773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d: Bootstrap starting.
I20260619 01:09:33.598973 12773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:33.600013 12773 log.cc:824] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d: Log is configured to *not* fsync() on all Append() calls
Jun 19 01:09:33 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:33.603410 12773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d: No bootstrap required, opened a new log
I20260619 01:09:33.610198 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:33.589369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.0.97.62:34645)
0619 01:09:33.590251 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:33.590264 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:33.590309 (+ 45us) server_negotiation.cc:407] Connection header received
0619 01:09:33.591998 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:33.592021 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:33.592480 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:33.592923 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:33.594323 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:33.595650 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:33.597138 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:33.597897 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:33.598097 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:33.602370 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:33.602378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:33.602393 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:33.602428 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:33.605613 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:33.606546 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:33.606565 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:33.606590 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:33.606836 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:33.607308 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:33.607323 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:33.607325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:33.607865 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:33.608093 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:33.608601 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 01:09:33.608969 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":415,"threads_started":1}
I20260619 01:09:33.612510 12773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } }
I20260619 01:09:33.612898 12773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:33.613019 12773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01bea5ad3bca4eb49d036006597d8d0d, State: Initialized, Role: FOLLOWER
I20260619 01:09:33.613662 12773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } }
I20260619 01:09:33.613909 12773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:09:33.614053 12773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:09:33.614230 12773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:33.616384 12773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } }
I20260619 01:09:33.616890 12773 leader_election.cc:302] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01bea5ad3bca4eb49d036006597d8d0d; no voters:
I20260619 01:09:33.617552 12773 leader_election.cc:288] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:09:33.617681 12778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:09:33.618999 12778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 1 LEADER]: Becoming Leader. State: Replica: 01bea5ad3bca4eb49d036006597d8d0d, State: Running, Role: LEADER
I20260619 01:09:33.619656 12778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } }
I20260619 01:09:33.619966 12773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:09:33.622576 12780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01bea5ad3bca4eb49d036006597d8d0d. Latest consensus state: current_term: 1 leader_uuid: "01bea5ad3bca4eb49d036006597d8d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } } }
I20260619 01:09:33.623104 12780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:33.625427 12779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01bea5ad3bca4eb49d036006597d8d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } } }
I20260619 01:09:33.626034 12779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:33.627499 12783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:09:33.632638 12783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:09:33.633451 12773 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 01:09:33.651468 12783 catalog_manager.cc:1347] Generated new cluster ID: d7f2c2471875475fbefe32fe0d4a1725
I20260619 01:09:33.651747 12783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:09:33.672654 12783 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:09:33.674185 12783 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:09:33.684747 12783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d: Generated new TSK 0
I20260619 01:09:33.686005 12783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:09:34.438287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34645
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:09:34 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781831374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 01:09:34.880918 12818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:34.881363 12818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:34.881588 12818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:34.893456 12818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:34.893612 12818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:34.893846 12818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:09:34.909762 12818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:40457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:34.911690 12818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:34.913465 12818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:34.936415 12829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:34.937041 12828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:34.942629 12818 server_base.cc:1034] running on GCE node
W20260619 01:09:34.946049 12831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:34.946970 12818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:34.948233 12818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:34.949479 12818 hybrid_clock.cc:648] HybridClock initialized: now 1781831374949401 us; error 64 us; skew 500 ppm
I20260619 01:09:34.949903 12818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:34 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:34.954056 12818 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:09:34.959201 12818 webserver.cc:460] Webserver started at http://127.0.97.1:37533/ using document root <none> and password file <none>
I20260619 01:09:34.959964 12818 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:34.960088 12818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:34.960403 12818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:09:34 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:09:34.982730 12818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "434a8706fea84a218dde406d5cebc674"
format_stamp: "Formatted at 2026-06-19 01:09:34 on dist-test-slave-5p8w"
server_key: "3555eb588a877ebfa47264501f5b0c21"
server_key_iv: "084d76e24d4c8466fcb82703b8be3316"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:34.983901 12818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "434a8706fea84a218dde406d5cebc674"
format_stamp: "Formatted at 2026-06-19 01:09:34 on dist-test-slave-5p8w"
server_key: "3555eb588a877ebfa47264501f5b0c21"
server_key_iv: "084d76e24d4c8466fcb82703b8be3316"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:35.005059 12818 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.007s sys 0.012s
I20260619 01:09:35.043191 12838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:35.046731 12818 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.000s sys 0.012s
I20260619 01:09:35.047223 12818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "434a8706fea84a218dde406d5cebc674"
format_stamp: "Formatted at 2026-06-19 01:09:34 on dist-test-slave-5p8w"
server_key: "3555eb588a877ebfa47264501f5b0c21"
server_key_iv: "084d76e24d4c8466fcb82703b8be3316"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:35.047629 12818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:35.082337 12818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:35.088338 12818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:35.088783 12818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:35.090456 12818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:35.093654 12818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:35.093833 12818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:35.094054 12818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:35.094197 12818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:35.151093 12818 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36483
I20260619 01:09:35.151086 12950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36483 every 8 connection(s)
I20260619 01:09:35.153040 12818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:09:35.163913 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 12818
I20260619 01:09:35.164132 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:09:35.164934 388 external_mini_cluster.cc:1428] Setting key 1f7fc172a0ad54958e584e7a3571260b
Jun 19 01:09:35 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 01:09:35 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:35.225394 12953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:35.162266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34339 (local address 127.0.97.62:34645)
0619 01:09:35.180979 (+ 18713us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:35.180986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:35.181014 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:09:35.182577 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:35.182583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:35.182687 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:35.187782 (+ 5095us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:35.205456 (+ 17674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:35.206647 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:35.208625 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:35.209231 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:35.210865 (+ 1634us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:35.215899 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:35.215906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:35.215910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:35.215953 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:35.219379 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:35.220704 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:35.220713 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:35.220720 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:35.221084 (+ 364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:35.221946 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:35.221952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:35.221956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:35.223571 (+ 1615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:35.223828 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:35.224695 (+ 867us) server_negotiation.cc:299] Negotiation successful
0619 01:09:35.224953 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18548,"thread_start_us":19019,"threads_started":1}
I20260619 01:09:35.226564 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:35.162993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34645 (local address 127.0.97.1:34339)
0619 01:09:35.178057 (+ 15064us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:35.178118 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:35.182027 (+ 3909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:35.189178 (+ 7151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:35.189194 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:35.189974 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:35.197742 (+ 7768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:35.197781 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:35.206927 (+ 9146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:35.206934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:35.208442 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:35.208461 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:35.211202 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:35.211210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:35.211550 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:35.212631 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:35.212666 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:35.215527 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:35.219584 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:35.219594 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:35.219616 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:35.220529 (+ 913us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:35.221447 (+ 918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:35.221452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:35.221455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:35.221793 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:35.223886 (+ 2093us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:35.223896 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:35.224280 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 01:09:35.224590 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 01:09:35.225200 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14702,"thread_start_us":14404,"threads_started":1}
I20260619 01:09:35.228608 12951 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34645
I20260619 01:09:35.229007 12951 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:35.230122 12951 heartbeater.cc:511] Master 127.0.97.62:34645 requested a full tablet report, sending...
I20260619 01:09:35.232930 12738 ts_manager.cc:194] Registered new tserver with Master: 434a8706fea84a218dde406d5cebc674 (127.0.97.1:36483)
I20260619 01:09:35.235077 12738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34339
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260619 01:09:35.291558 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.107s user 0.002s sys 0.018s
I20260619 01:09:35.292493 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34645
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:09:35 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 01:09:35.761768 12957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:35.762195 12957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:35.762406 12957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:35.774356 12957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:35.774611 12957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:35.774847 12957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:09:35.789623 12957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:40457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:35.791074 12957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:35.792524 12957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:35.809814 12974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:35.809864 12971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:35.809837 12972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:35.811084 12957 server_base.cc:1034] running on GCE node
I20260619 01:09:35.812089 12957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:35.813156 12957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:35.814394 12957 hybrid_clock.cc:648] HybridClock initialized: now 1781831375814352 us; error 64 us; skew 500 ppm
I20260619 01:09:35.814742 12957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:35 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:35.825965 12957 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:09:35.827713 12957 webserver.cc:460] Webserver started at http://127.0.97.2:33463/ using document root <none> and password file <none>
I20260619 01:09:35.828507 12957 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:35.828672 12957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:35.829025 12957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:09:35 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:09:35.844272 12957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "58c573862fc945908bb2c05d045a0d8d"
format_stamp: "Formatted at 2026-06-19 01:09:35 on dist-test-slave-5p8w"
server_key: "df00776543e9a150100001fd16a55e6c"
server_key_iv: "ba107b5fc35c397ca951e6d13054bd5b"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:35.845211 12957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "58c573862fc945908bb2c05d045a0d8d"
format_stamp: "Formatted at 2026-06-19 01:09:35 on dist-test-slave-5p8w"
server_key: "df00776543e9a150100001fd16a55e6c"
server_key_iv: "ba107b5fc35c397ca951e6d13054bd5b"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:35.851223 12957 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 01:09:35.866356 12981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:35.868283 12957 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:09:35.868508 12957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "58c573862fc945908bb2c05d045a0d8d"
format_stamp: "Formatted at 2026-06-19 01:09:35 on dist-test-slave-5p8w"
server_key: "df00776543e9a150100001fd16a55e6c"
server_key_iv: "ba107b5fc35c397ca951e6d13054bd5b"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:35.868774 12957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:35.891809 12957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:35.896374 12957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:35.896703 12957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:35.898029 12957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:35.899556 12957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:35.899668 12957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:35.899825 12957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:35.899904 12957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:35.937162 12957 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36301
I20260619 01:09:35.937181 13093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36301 every 8 connection(s)
I20260619 01:09:35.938678 12957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:09:35.948293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 12957
I20260619 01:09:35.948539 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:09:35.949131 388 external_mini_cluster.cc:1428] Setting key f52a5d4f69c38b7a3a2a2bd73c8f7446
Jun 19 01:09:35 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:35.964897 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:35.942124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34645 (local address 127.0.97.2:45929)
0619 01:09:35.943013 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:35.943060 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:35.944703 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:35.945572 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:35.945581 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:35.946036 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:35.946635 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:35.946652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:35.948172 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:35.948178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:35.950202 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:35.950218 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:35.951322 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:35.951330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:35.951654 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:35.952486 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:35.952525 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:35.955588 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:35.960336 (+ 4748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:35.960370 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:35.960396 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:35.961138 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:35.961706 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:35.961713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:35.961719 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:35.962009 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:35.962803 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:35.962815 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:35.963215 (+ 400us) client_negotiation.cc:769] Sending connection context
0619 01:09:35.963447 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 01:09:35.963854 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":384,"threads_started":1}
I20260619 01:09:35.966308 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:35.941584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45929 (local address 127.0.97.62:34645)
0619 01:09:35.942097 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:35.942103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:35.943208 (+ 1105us) server_negotiation.cc:407] Connection header received
0619 01:09:35.944990 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:35.944997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:35.945137 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:35.945276 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:35.946850 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:35.947971 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:35.950375 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:35.951064 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:35.951209 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:35.955925 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:35.955935 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:35.955940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:35.956073 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:35.960144 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:35.961334 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:35.961344 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:35.961351 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:35.961544 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:35.962179 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:35.962185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:35.962190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:35.962634 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:35.965751 (+ 3117us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:35.965860 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 01:09:35.966052 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":243,"threads_started":1}
I20260619 01:09:35.968463 13094 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34645
I20260619 01:09:35.968832 13094 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:35.969759 13094 heartbeater.cc:511] Master 127.0.97.62:34645 requested a full tablet report, sending...
I20260619 01:09:35.971422 12738 ts_manager.cc:194] Registered new tserver with Master: 58c573862fc945908bb2c05d045a0d8d (127.0.97.2:36301)
I20260619 01:09:35.972301 12738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45929
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:09:36.009130 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34645
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:09:36.239559 12951 heartbeater.cc:503] Master 127.0.97.62:34645 was elected leader, sending a full tablet report...
W20260619 01:09:36.377525 13100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:36.377817 13100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:36.377907 13100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:36.385167 13100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:36.385327 13100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:36.385442 13100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:09:36.395495 13100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:40457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:36.397075 13100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:36.398388 13100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:36.411722 13105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:36.411671 13106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:36.411746 13108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:36.412508 13100 server_base.cc:1034] running on GCE node
I20260619 01:09:36.413121 13100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:36.414131 13100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:36.415328 13100 hybrid_clock.cc:648] HybridClock initialized: now 1781831376415285 us; error 38 us; skew 500 ppm
I20260619 01:09:36.415649 13100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:36 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:36.418802 13100 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:09:36.420265 13100 webserver.cc:460] Webserver started at http://127.0.97.3:36881/ using document root <none> and password file <none>
I20260619 01:09:36.420926 13100 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:36.421100 13100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:36.421490 13100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:09:36 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:09:36.436165 13100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "629558fb276246329bbb881555c4ade0"
format_stamp: "Formatted at 2026-06-19 01:09:36 on dist-test-slave-5p8w"
server_key: "3b96de7f301445c44100b488180454fb"
server_key_iv: "e245eb4362665edeb0eec3545f70865a"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:36.437067 13100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "629558fb276246329bbb881555c4ade0"
format_stamp: "Formatted at 2026-06-19 01:09:36 on dist-test-slave-5p8w"
server_key: "3b96de7f301445c44100b488180454fb"
server_key_iv: "e245eb4362665edeb0eec3545f70865a"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:36.442739 13100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 01:09:36.460515 13115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:36.462535 13100 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:09:36.462785 13100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "629558fb276246329bbb881555c4ade0"
format_stamp: "Formatted at 2026-06-19 01:09:36 on dist-test-slave-5p8w"
server_key: "3b96de7f301445c44100b488180454fb"
server_key_iv: "e245eb4362665edeb0eec3545f70865a"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:36.463066 13100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:36.480901 13100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:36.486390 13100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:36.486789 13100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:36.488056 13100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:36.490036 13100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:36.490160 13100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:36.490305 13100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:36.490366 13100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:36.525532 13100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34807
I20260619 01:09:36.525554 13227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34807 every 8 connection(s)
I20260619 01:09:36.526988 13100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:09:36.536271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 13100
I20260619 01:09:36.536512 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:09:36.536985 388 external_mini_cluster.cc:1428] Setting key 11bcf4551a3e6fee6b2a9ea2322e7ed1
Jun 19 01:09:36 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:36.549548 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:36.529670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47469 (local address 127.0.97.62:34645)
0619 01:09:36.530229 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:36.530234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:36.531335 (+ 1101us) server_negotiation.cc:407] Connection header received
0619 01:09:36.533244 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:36.533250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:36.533380 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:36.533503 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:36.535439 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:36.536189 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:36.538269 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:36.538817 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:36.538981 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:36.542697 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:36.542703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:36.542706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:36.542743 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:36.545691 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:36.546596 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:36.546601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:36.546604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:36.546760 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:36.547310 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:36.547315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:36.547317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:36.547664 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:36.547813 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:36.548921 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0619 01:09:36.549156 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":260,"threads_started":1}
I20260619 01:09:36.550668 13231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:36.530343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34645 (local address 127.0.97.3:47469)
0619 01:09:36.531150 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:36.531193 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:36.532918 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:36.533844 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:36.533854 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:36.534531 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:36.535218 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:36.535239 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:36.536390 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:36.536395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:36.538116 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:36.538130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:36.539096 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:36.539105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:36.539403 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:36.540212 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:36.540244 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:36.542431 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:36.545868 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:36.545877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:36.545893 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:36.546401 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:36.546941 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:36.546945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:36.546948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:36.547168 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:36.548000 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:36.548025 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:36.548465 (+ 440us) client_negotiation.cc:769] Sending connection context
0619 01:09:36.548782 (+ 317us) client_negotiation.cc:240] Negotiation successful
0619 01:09:36.549251 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":350,"threads_started":1}
I20260619 01:09:36.552453 13228 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34645
I20260619 01:09:36.552803 13228 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:36.553697 13228 heartbeater.cc:511] Master 127.0.97.62:34645 requested a full tablet report, sending...
I20260619 01:09:36.555284 12737 ts_manager.cc:194] Registered new tserver with Master: 629558fb276246329bbb881555c4ade0 (127.0.97.3:34807)
I20260619 01:09:36.556388 12737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47469
I20260619 01:09:36.565956 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:09:36.567912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 12818
Jun 19 01:09:36 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:09:36.586377 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:36.569061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.0.97.1:36483)
0619 01:09:36.569857 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:36.569867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:36.569899 (+ 32us) server_negotiation.cc:407] Connection header received
0619 01:09:36.570030 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:36.570039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:36.570314 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:36.570506 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:36.572122 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:36.573186 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:36.574669 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:36.575252 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:36.575401 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:36.580113 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:36.580127 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:36.580155 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:36.580205 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:36.583361 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:36.584153 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:36.584161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:36.584163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:36.584325 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:36.584729 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:36.584732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:36.584735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:36.585082 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:36.585435 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:36.585736 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 01:09:36.586018 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":433,"threads_started":1}
I20260619 01:09:36.587911 12946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:36.673935 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 12818
I20260619 01:09:36.689865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 12957
Jun 19 01:09:36 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:09:36.712301 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:36.690852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.0.97.2:36301)
0619 01:09:36.691457 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:36.691465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:36.691489 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:09:36.691550 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:36.691556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:36.691912 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:36.692088 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:36.693466 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:36.694482 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:36.695944 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:36.696546 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:36.696716 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:36.705798 (+ 9082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:36.705816 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:36.705832 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:36.705876 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:36.708972 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:36.709934 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:36.709943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:36.709945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:36.710096 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:36.710583 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:36.710587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:36.710595 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:36.710972 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:36.711303 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:36.711659 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 01:09:36.711942 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":355,"threads_started":1}
I20260619 01:09:36.713770 13089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:36.797282 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 12957
I20260619 01:09:36.813542 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 13100
Jun 19 01:09:36 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:09:36.831288 13237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:36.814510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.0.97.3:34807)
0619 01:09:36.815208 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:36.815216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:36.815244 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:09:36.815305 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:36.815312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:36.815539 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:36.815695 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:36.817084 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:36.818224 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:36.819559 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:36.820160 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:36.820336 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:36.825225 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:36.825241 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:36.825258 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:36.825364 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:36.828416 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:36.829241 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:36.829249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:36.829251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:36.829450 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:36.829916 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:36.829919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:36.829921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:36.830307 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:36.830567 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:36.830787 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 01:09:36.830987 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":361,"threads_started":1}
I20260619 01:09:36.832603 13223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:36.917032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 13100
I20260619 01:09:36.933060 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 12706
I20260619 01:09:36.934361 12768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:37.046440 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 12706
I20260619 01:09:37.068182 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34645
--webserver_interface=127.0.97.62
--webserver_port=39567
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34645
--ranger_config_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40457/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40457/kms with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:09:37.426965 13240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:37.427325 13240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:37.427520 13240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:37.435936 13240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:09:37.436082 13240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:37.436192 13240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:37.436290 13240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:09:37.436364 13240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:09:37.447333 13240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40457/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34645
--ranger_config_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:37.448868 13240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:37.450487 13240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:37.464280 13245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:37.464313 13246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:37.464247 13248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:37.464905 13240 server_base.cc:1034] running on GCE node
I20260619 01:09:37.465902 13240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:37.467542 13240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:37.468771 13240 hybrid_clock.cc:648] HybridClock initialized: now 1781831377468714 us; error 51 us; skew 500 ppm
I20260619 01:09:37.469190 13240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:37 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:37.472498 13240 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:09:37.474251 13240 webserver.cc:460] Webserver started at http://127.0.97.62:39567/ using document root <none> and password file <none>
I20260619 01:09:37.474995 13240 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:37.475152 13240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 01:09:37 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:09:37.559218 13240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:09:37.562563 13256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:37.564245 13240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:09:37.564479 13240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "01bea5ad3bca4eb49d036006597d8d0d"
format_stamp: "Formatted at 2026-06-19 01:09:33 on dist-test-slave-5p8w"
server_key: "4b7f659455b4830af34a31fd29b7887b"
server_key_iv: "5c833c4c97d22aec385d54fd187fb2ca"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:37.565608 13240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:37.580518 13240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:37.585158 13240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:37.585526 13240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:37.604496 13240 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34645
I20260619 01:09:37.604503 13307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34645 every 8 connection(s)
I20260619 01:09:37.606101 13240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:09:37.612891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 13240
I20260619 01:09:37.613783 13308 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:09:37.614015 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36483
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34645
--webserver_port=37533
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40457/kms with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:09:37.619546 13308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d: Bootstrap starting.
I20260619 01:09:37.639772 13308 log.cc:824] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:37.648830 13308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:09:37.649523 13308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d: Bootstrap complete.
I20260619 01:09:37.658876 13308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } }
I20260619 01:09:37.659807 13308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01bea5ad3bca4eb49d036006597d8d0d, State: Initialized, Role: FOLLOWER
I20260619 01:09:37.660480 13308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } }
I20260619 01:09:37.660717 13308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:09:37.660864 13308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:09:37.661055 13308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:09:37.665156 13308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } }
I20260619 01:09:37.665791 13308 leader_election.cc:302] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01bea5ad3bca4eb49d036006597d8d0d; no voters:
I20260619 01:09:37.666486 13308 leader_election.cc:288] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 01:09:37.666668 13312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 01:09:37.667857 13312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [term 2 LEADER]: Becoming Leader. State: Replica: 01bea5ad3bca4eb49d036006597d8d0d, State: Running, Role: LEADER
I20260619 01:09:37.668584 13312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } }
I20260619 01:09:37.668969 13308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:09:37.671654 13314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01bea5ad3bca4eb49d036006597d8d0d. Latest consensus state: current_term: 2 leader_uuid: "01bea5ad3bca4eb49d036006597d8d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } } }
I20260619 01:09:37.672089 13314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:37.671602 13313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "01bea5ad3bca4eb49d036006597d8d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01bea5ad3bca4eb49d036006597d8d0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34645 } } }
I20260619 01:09:37.672286 13313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:37.673121 13319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:09:37.682734 13319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:09:37.684360 13319 catalog_manager.cc:1259] Loaded cluster ID: d7f2c2471875475fbefe32fe0d4a1725
I20260619 01:09:37.684525 13319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:09:37.689658 13319 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:09:37.692077 13319 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d: Loaded TSK: 0
I20260619 01:09:37.694492 13319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:09:37.699093 13308 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 01:09:38.088532 13310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:38.088986 13310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:38.089218 13310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:38.101502 13310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:38.101670 13310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:38.101881 13310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:09:38.118408 13310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37533
--tserver_master_addrs=127.0.97.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:38.120527 13310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:38.122503 13310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:38.163177 13346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:38.165828 13345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:38.174662 13348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:38.178594 13310 server_base.cc:1034] running on GCE node
I20260619 01:09:38.179522 13310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:38.193667 13310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:38.197243 13310 hybrid_clock.cc:648] HybridClock initialized: now 1781831378197078 us; error 167 us; skew 500 ppm
I20260619 01:09:38.197854 13310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:38 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:38.203116 13310 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:09:38.215215 13310 webserver.cc:460] Webserver started at http://127.0.97.1:37533/ using document root <none> and password file <none>
I20260619 01:09:38.216166 13310 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:38.216377 13310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 01:09:38 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:09:38.288241 13310 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.004s sys 0.000s
I20260619 01:09:38.296406 13357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:38.298547 13310 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 01:09:38.298763 13310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "434a8706fea84a218dde406d5cebc674"
format_stamp: "Formatted at 2026-06-19 01:09:34 on dist-test-slave-5p8w"
server_key: "3555eb588a877ebfa47264501f5b0c21"
server_key_iv: "084d76e24d4c8466fcb82703b8be3316"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:38.300233 13310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:38.324281 13310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:38.329380 13310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:38.329733 13310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:38.331524 13310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:38.339691 13310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:38.339890 13310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 01:09:38.340083 13310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:38.340217 13310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:38.404040 13310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36483
I20260619 01:09:38.404138 13469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36483 every 8 connection(s)
I20260619 01:09:38.406154 13310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:09:38.409689 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 13310
I20260619 01:09:38.410521 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36301
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34645
--webserver_port=33463
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40457/kms with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:09:38 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:38.462577 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:38.412969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34645 (local address 127.0.97.1:39147)
0619 01:09:38.433833 (+ 20864us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:38.433906 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:38.437338 (+ 3432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:38.439627 (+ 2289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:38.439639 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:38.440325 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:38.441155 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:38.441187 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:38.443827 (+ 2640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:38.443833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:38.445993 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:38.446010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:38.448002 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:38.448011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:38.448325 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:38.449233 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:38.449268 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:38.453398 (+ 4130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:38.457186 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:38.457196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:38.457216 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:38.457922 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:38.458543 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:38.458548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:38.458552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:38.458830 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:38.459869 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:38.459878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:38.460244 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 01:09:38.460516 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 01:09:38.461123 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20453,"thread_start_us":20156,"threads_started":1}
I20260619 01:09:38.462577 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:38.418509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39147 (local address 127.0.97.62:34645)
0619 01:09:38.426900 (+ 8391us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:38.426919 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:38.434710 (+ 7791us) server_negotiation.cc:407] Connection header received
0619 01:09:38.438241 (+ 3531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:38.438270 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:38.438902 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:38.439434 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:38.441481 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:38.443555 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:38.446216 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:38.447363 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:38.447572 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:38.453754 (+ 6182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:38.453765 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:38.453784 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:38.453830 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:38.456984 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:38.458133 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:38.458143 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:38.458152 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:38.458378 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:38.458980 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:38.458985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:38.458989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:38.459725 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:38.459944 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:38.460766 (+ 822us) server_negotiation.cc:299] Negotiation successful
0619 01:09:38.461143 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8101,"thread_start_us":7840,"threads_started":1}
I20260619 01:09:38.466526 13470 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34645
I20260619 01:09:38.466925 13470 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:38.467958 13470 heartbeater.cc:511] Master 127.0.97.62:34645 requested a full tablet report, sending...
W20260619 01:09:38.471589 13470 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34645 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 01:09:38.471848 13470 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:38.472184 13470 heartbeater.cc:511] Master 127.0.97.62:34645 requested a full tablet report, sending...
I20260619 01:09:38.474125 13470 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:38.474442 13470 heartbeater.cc:511] Master 127.0.97.62:34645 requested a full tablet report, sending...
W20260619 01:09:38.475700 13470 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:09:38.937282 13474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:38.937667 13474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:38.937841 13474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:38.945686 13474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:38.945873 13474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:38.946070 13474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:09:38.956626 13474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33463
--tserver_master_addrs=127.0.97.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:38.958211 13474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:38.959544 13474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:38.978289 13490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:38.979235 13491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:38.983196 13493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:38.983762 13474 server_base.cc:1034] running on GCE node
I20260619 01:09:38.984589 13474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:38.986321 13474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:38.987629 13474 hybrid_clock.cc:648] HybridClock initialized: now 1781831378987562 us; error 58 us; skew 500 ppm
I20260619 01:09:38.987985 13474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:38 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:38.991742 13474 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:09:38.993687 13474 webserver.cc:460] Webserver started at http://127.0.97.2:33463/ using document root <none> and password file <none>
Jun 19 01:09:38 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781831378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:38.994596 13474 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:38.994781 13474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 01:09:38 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:09:39.018733 13474 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 01:09:39.021780 13501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:39.023329 13474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:09:39.023532 13474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "58c573862fc945908bb2c05d045a0d8d"
format_stamp: "Formatted at 2026-06-19 01:09:35 on dist-test-slave-5p8w"
server_key: "df00776543e9a150100001fd16a55e6c"
server_key_iv: "ba107b5fc35c397ca951e6d13054bd5b"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:39.024554 13474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:39.063849 13474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:39.068835 13474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:39.069175 13474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:39.070469 13474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:39.072073 13474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:39.072191 13474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:39.072324 13474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:39.072429 13474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:39.150954 13474 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36301
I20260619 01:09:39.151055 13614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36301 every 8 connection(s)
I20260619 01:09:39.152971 13474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:09:39.161882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 13474
I20260619 01:09:39.162722 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:40457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34807
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34645
--webserver_port=36881
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40457/kms with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:09:39 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:39.191756 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:39.158119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34645 (local address 127.0.97.2:51657)
0619 01:09:39.160625 (+ 2506us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:39.160684 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:39.163475 (+ 2791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:39.166947 (+ 3472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:39.166960 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:39.167730 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:39.170683 (+ 2953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:39.170725 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:39.172496 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:39.172502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:39.175035 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:39.175057 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:39.176640 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:39.176650 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:39.177048 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:39.177933 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:39.177970 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:39.180864 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:39.185858 (+ 4994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:39.185872 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:39.185900 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:39.186665 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:39.187227 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:39.187235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:39.187240 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:39.187535 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:39.188403 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:39.188415 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:39.188961 (+ 546us) client_negotiation.cc:769] Sending connection context
0619 01:09:39.189363 (+ 402us) client_negotiation.cc:240] Negotiation successful
0619 01:09:39.190058 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2142,"thread_start_us":1880,"threads_started":1}
I20260619 01:09:39.193215 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:39.158855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51657 (local address 127.0.97.62:34645)
0619 01:09:39.166191 (+ 7336us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:39.166201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:39.166231 (+ 30us) server_negotiation.cc:407] Connection header received
0619 01:09:39.166311 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:39.166321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:39.166431 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:39.166582 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:39.171014 (+ 4432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:39.172296 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:39.175334 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:39.176123 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:39.176272 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:39.181200 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:39.181210 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:39.181218 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:39.181268 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:39.185624 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:39.186865 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:39.186873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:39.186879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:39.187066 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:39.187691 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:39.187700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:39.187705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:39.188153 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:39.188319 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:39.192401 (+ 4082us) server_negotiation.cc:299] Negotiation successful
0619 01:09:39.192707 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7157,"thread_start_us":7025,"threads_started":1}
I20260619 01:09:39.195585 13615 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34645
I20260619 01:09:39.195972 13615 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:39.196949 13615 heartbeater.cc:511] Master 127.0.97.62:34645 requested a full tablet report, sending...
I20260619 01:09:39.199041 13273 ts_manager.cc:194] Registered new tserver with Master: 58c573862fc945908bb2c05d045a0d8d (127.0.97.2:36301)
I20260619 01:09:39.200606 13273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51657
I20260619 01:09:39.475868 13470 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:39.476351 13470 heartbeater.cc:511] Master 127.0.97.62:34645 requested a full tablet report, sending...
I20260619 01:09:39.477737 13273 ts_manager.cc:194] Registered new tserver with Master: 434a8706fea84a218dde406d5cebc674 (127.0.97.1:36483)
I20260619 01:09:39.478960 13273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39147
Jun 19 01:09:39 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 01:09:39.649211 13619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:39.649679 13619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:39.649892 13619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:39.661787 13619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:39.662034 13619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:39.662284 13619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:09:39.678723 13619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:40457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36881
--tserver_master_addrs=127.0.97.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:39.680608 13619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:39.682294 13619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:39.699120 13624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:39.707842 13625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:39.709348 13627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:39.710647 13619 server_base.cc:1034] running on GCE node
I20260619 01:09:39.711544 13619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:39.712980 13619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:39.714336 13619 hybrid_clock.cc:648] HybridClock initialized: now 1781831379714235 us; error 89 us; skew 500 ppm
I20260619 01:09:39.714746 13619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:39 dist-test-slave-5p8w krb5kdc[12262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:39.719318 13619 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:09:39.721551 13619 webserver.cc:460] Webserver started at http://127.0.97.3:36881/ using document root <none> and password file <none>
I20260619 01:09:39.722618 13619 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:39.722803 13619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 01:09:39 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:09:39.755421 13619 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
I20260619 01:09:39.760775 13635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:39.762845 13619 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260619 01:09:39.763058 13619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "629558fb276246329bbb881555c4ade0"
format_stamp: "Formatted at 2026-06-19 01:09:36 on dist-test-slave-5p8w"
server_key: "3b96de7f301445c44100b488180454fb"
server_key_iv: "e245eb4362665edeb0eec3545f70865a"
server_key_version: "kuduclusterkey@0"
I20260619 01:09:39.764585 13619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:39.785137 13619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:39.790639 13619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:39.790987 13619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:39.798835 13619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:39.801059 13619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:39.801275 13619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 01:09:39.801486 13619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:39.801620 13619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:09:39 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:09:40.043097 13619 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34807
I20260619 01:09:40.043170 13748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34807 every 8 connection(s)
I20260619 01:09:40.045158 13619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:09:40.049839 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 13619
Jun 19 01:09:40 dist-test-slave-5p8w krb5kdc[12262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:40.102146 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.049260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34645 (local address 127.0.97.3:40325)
0619 01:09:40.060050 (+ 10790us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.060123 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:40.068594 (+ 8471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:40.079151 (+ 10557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:40.079170 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:40.079924 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:40.080926 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.080958 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.082864 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.082870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.085222 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.085245 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.086764 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.086772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.087123 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.088533 (+ 1410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:40.088566 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:40.091753 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:40.096459 (+ 4706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:40.096471 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:40.096495 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:40.097244 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:40.097854 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:40.097860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:40.097864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:40.098166 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:40.098962 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:40.098971 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:40.099498 (+ 527us) client_negotiation.cc:769] Sending connection context
0619 01:09:40.099837 (+ 339us) client_negotiation.cc:240] Negotiation successful
0619 01:09:40.100474 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5572,"thread_start_us":5231,"threads_started":1}
I20260619 01:09:40.103420 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.050923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40325 (local address 127.0.97.62:34645)
0619 01:09:40.077700 (+ 26777us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.077707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.077739 (+ 32us) server_negotiation.cc:407] Connection header received
0619 01:09:40.077821 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.077827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.077939 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.081230 (+ 3291us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:40.081455 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.082640 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.085554 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.086252 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.086402 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.092093 (+ 5691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:40.092101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:40.092106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:40.092165 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:40.096244 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.097475 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.097480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.097484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.097680 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.098319 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.098324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.098328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.098719 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:40.098870 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.102715 (+ 3845us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.102990 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26607,"spinlock_wait_cycles":13056,"thread_start_us":36621,"threads_started":1}
I20260619 01:09:40.106316 13749 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34645
I20260619 01:09:40.106688 13749 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:40.107697 13749 heartbeater.cc:511] Master 127.0.97.62:34645 requested a full tablet report, sending...
I20260619 01:09:40.109545 13273 ts_manager.cc:194] Registered new tserver with Master: 629558fb276246329bbb881555c4ade0 (127.0.97.3:34807)
I20260619 01:09:40.110162 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.051395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.0.97.62:34645)
0619 01:09:40.078648 (+ 27253us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.078654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.088316 (+ 9662us) server_negotiation.cc:407] Connection header received
0619 01:09:40.090359 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.090368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.090474 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.090603 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:40.092845 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.094361 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.096530 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.097377 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.097554 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.101539 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:40.101548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:40.101557 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:40.101605 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:40.105963 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.107275 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.107282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.107285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.107457 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.107978 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.107985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.107990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.108398 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:40.109258 (+ 860us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.109437 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.109700 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27116,"thread_start_us":26749,"threads_started":1}
I20260619 01:09:40.110785 13273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40325
I20260619 01:09:40.113131 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:09:40.130849 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.116567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.0.97.62:34645)
0619 01:09:40.116796 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.116804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.117171 (+ 367us) server_negotiation.cc:407] Connection header received
0619 01:09:40.117451 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.117455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.117521 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.117639 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:40.119368 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.120108 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.121602 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.122120 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.122260 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.125041 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:40.125047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:40.125050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:40.125088 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:40.127922 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.128710 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.128716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.128720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.128917 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.129506 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.129511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.129513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.129784 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:40.130075 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.130424 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.130611 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:09:40.136776 13273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:09:40.144480 13273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:09:40.201862 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.162676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34807 (local address 127.0.0.1:37064)
0619 01:09:40.187253 (+ 24577us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.187311 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:40.187580 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:40.189860 (+ 2280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:40.189872 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:40.189926 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:40.190536 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.190552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.192923 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.192929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.195868 (+ 2939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.195888 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.200078 (+ 4190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.200085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.200325 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.200385 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 01:09:40.200538 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 01:09:40.200755 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24227,"thread_start_us":27301,"threads_started":1}
I20260619 01:09:40.201900 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.163724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.0.97.3:34807)
0619 01:09:40.188722 (+ 24998us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.188730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.188768 (+ 38us) server_negotiation.cc:407] Connection header received
0619 01:09:40.188878 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.188886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.189202 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.189468 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:40.191088 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.192717 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.197872 (+ 5155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.199494 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.199636 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.199904 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.201336 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.201531 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24819,"thread_start_us":25600,"threads_started":1}
I20260619 01:09:40.203224 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.163196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.0.97.2:36301)
0619 01:09:40.188261 (+ 25065us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.188270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.192299 (+ 4029us) server_negotiation.cc:407] Connection header received
0619 01:09:40.192474 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.192487 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.192861 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.193176 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:40.194915 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.196838 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.199553 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.201103 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.201227 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.201500 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.202667 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.202860 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24879,"thread_start_us":26418,"threads_started":1}
I20260619 01:09:40.203712 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.162907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36301 (local address 127.0.0.1:44528)
0619 01:09:40.190308 (+ 27401us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.190330 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:40.190536 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:40.193562 (+ 3026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:40.193570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:40.193616 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:40.194564 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.194585 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.197015 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.197035 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.199345 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.199361 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.201650 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.201656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.201866 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.201910 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 01:09:40.201991 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 01:09:40.202172 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27239,"mutex_wait_us":3,"thread_start_us":33189,"threads_started":1}
I20260619 01:09:40.206837 13684 tablet_service.cc:1467] Processing CreateTablet for tablet f161a766bbfc4592bb74ec85a3667962 (DEFAULT_TABLE table=test-table [id=91fbe03ce5194afea181920cac1ff141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:09:40.207814 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.163282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36483 (local address 127.0.0.1:40170)
0619 01:09:40.196432 (+ 33150us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.196468 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:40.196735 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:40.197776 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:40.197782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:40.197815 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:40.198222 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.198235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.201820 (+ 3585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.201825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.204526 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.204546 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.206897 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.206906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.207107 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.207143 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 01:09:40.207249 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 01:09:40.207393 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32985,"mutex_wait_us":602,"thread_start_us":27420,"threads_started":1}
I20260619 01:09:40.207834 13615 heartbeater.cc:503] Master 127.0.97.62:34645 was elected leader, sending a full tablet report...
I20260619 01:09:40.208582 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.163888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.0.97.1:36483)
0619 01:09:40.186285 (+ 22397us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.186296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.196600 (+ 10304us) server_negotiation.cc:407] Connection header received
0619 01:09:40.196966 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.196976 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.197322 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.197549 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:40.198462 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.200555 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.204773 (+ 4218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.206361 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.206502 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.206726 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.208066 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.208262 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22165,"thread_start_us":22411,"threads_started":1}
I20260619 01:09:40.208258 13550 tablet_service.cc:1467] Processing CreateTablet for tablet f161a766bbfc4592bb74ec85a3667962 (DEFAULT_TABLE table=test-table [id=91fbe03ce5194afea181920cac1ff141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:09:40.210248 13550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f161a766bbfc4592bb74ec85a3667962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:40.212438 13684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f161a766bbfc4592bb74ec85a3667962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:40.214722 13405 tablet_service.cc:1467] Processing CreateTablet for tablet f161a766bbfc4592bb74ec85a3667962 (DEFAULT_TABLE table=test-table [id=91fbe03ce5194afea181920cac1ff141]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:09:40.216753 13405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f161a766bbfc4592bb74ec85a3667962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:40.242363 13777 tablet_bootstrap.cc:492] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d: Bootstrap starting.
I20260619 01:09:40.246466 13778 tablet_bootstrap.cc:492] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: Bootstrap starting.
I20260619 01:09:40.247659 13777 tablet_bootstrap.cc:654] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:40.249209 13777 log.cc:824] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:40.251911 13778 tablet_bootstrap.cc:654] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:40.253834 13778 log.cc:824] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:40.259564 13777 tablet_bootstrap.cc:492] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d: No bootstrap required, opened a new log
I20260619 01:09:40.259994 13777 ts_tablet_manager.cc:1397] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.005s
I20260619 01:09:40.261679 13779 tablet_bootstrap.cc:492] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674: Bootstrap starting.
I20260619 01:09:40.263339 13778 tablet_bootstrap.cc:492] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: No bootstrap required, opened a new log
I20260619 01:09:40.263657 13778 ts_tablet_manager.cc:1397] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260619 01:09:40.265688 13779 tablet_bootstrap.cc:654] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:40.267093 13779 log.cc:824] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:40.273283 13778 raft_consensus.cc:348] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629558fb276246329bbb881555c4ade0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34807 } } peers { permanent_uuid: "58c573862fc945908bb2c05d045a0d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36301 } } peers { permanent_uuid: "434a8706fea84a218dde406d5cebc674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36483 } }
I20260619 01:09:40.273451 13777 raft_consensus.cc:348] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629558fb276246329bbb881555c4ade0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34807 } } peers { permanent_uuid: "58c573862fc945908bb2c05d045a0d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36301 } } peers { permanent_uuid: "434a8706fea84a218dde406d5cebc674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36483 } }
I20260619 01:09:40.273986 13777 raft_consensus.cc:374] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:40.274022 13778 raft_consensus.cc:374] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:40.274153 13777 raft_consensus.cc:729] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c573862fc945908bb2c05d045a0d8d, State: Initialized, Role: FOLLOWER
I20260619 01:09:40.274200 13778 raft_consensus.cc:729] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629558fb276246329bbb881555c4ade0, State: Initialized, Role: FOLLOWER
I20260619 01:09:40.275008 13778 consensus_queue.cc:260] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629558fb276246329bbb881555c4ade0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34807 } } peers { permanent_uuid: "58c573862fc945908bb2c05d045a0d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36301 } } peers { permanent_uuid: "434a8706fea84a218dde406d5cebc674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36483 } }
I20260619 01:09:40.275163 13777 consensus_queue.cc:260] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629558fb276246329bbb881555c4ade0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34807 } } peers { permanent_uuid: "58c573862fc945908bb2c05d045a0d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36301 } } peers { permanent_uuid: "434a8706fea84a218dde406d5cebc674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36483 } }
I20260619 01:09:40.282317 13779 tablet_bootstrap.cc:492] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674: No bootstrap required, opened a new log
I20260619 01:09:40.282637 13784 raft_consensus.cc:482] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:09:40.283051 13784 raft_consensus.cc:504] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629558fb276246329bbb881555c4ade0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34807 } } peers { permanent_uuid: "58c573862fc945908bb2c05d045a0d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36301 } } peers { permanent_uuid: "434a8706fea84a218dde406d5cebc674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36483 } }
I20260619 01:09:40.283375 13779 ts_tablet_manager.cc:1397] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.009s
I20260619 01:09:40.295883 13777 ts_tablet_manager.cc:1428] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d: Time spent starting tablet: real 0.036s user 0.016s sys 0.000s
I20260619 01:09:40.296424 13778 ts_tablet_manager.cc:1428] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: Time spent starting tablet: real 0.033s user 0.020s sys 0.013s
I20260619 01:09:40.299563 13784 leader_election.cc:288] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 629558fb276246329bbb881555c4ade0 (127.0.97.3:34807), 434a8706fea84a218dde406d5cebc674 (127.0.97.1:36483)
I20260619 01:09:40.301920 13749 heartbeater.cc:503] Master 127.0.97.62:34645 was elected leader, sending a full tablet report...
W20260619 01:09:40.303046 13750 tablet.cc:2369] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:09:40.304806 13779 raft_consensus.cc:348] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629558fb276246329bbb881555c4ade0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34807 } } peers { permanent_uuid: "58c573862fc945908bb2c05d045a0d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36301 } } peers { permanent_uuid: "434a8706fea84a218dde406d5cebc674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36483 } }
I20260619 01:09:40.305457 13779 raft_consensus.cc:374] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:40.305716 13779 raft_consensus.cc:729] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 434a8706fea84a218dde406d5cebc674, State: Initialized, Role: FOLLOWER
I20260619 01:09:40.306738 13779 consensus_queue.cc:260] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629558fb276246329bbb881555c4ade0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34807 } } peers { permanent_uuid: "58c573862fc945908bb2c05d045a0d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36301 } } peers { permanent_uuid: "434a8706fea84a218dde406d5cebc674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36483 } }
I20260619 01:09:40.311591 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.286282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33291 (local address 127.0.97.3:34807)
0619 01:09:40.286502 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.286509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.303601 (+ 17092us) server_negotiation.cc:407] Connection header received
0619 01:09:40.303904 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.303912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.303989 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.304053 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:40.304873 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.305942 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.308698 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.309946 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.310062 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.310196 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.311009 (+ 813us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.311216 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:09:40.314298 13470 heartbeater.cc:503] Master 127.0.97.62:34645 was elected leader, sending a full tablet report...
I20260619 01:09:40.315081 13779 ts_tablet_manager.cc:1428] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674: Time spent starting tablet: real 0.031s user 0.018s sys 0.002s
I20260619 01:09:40.315610 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.285795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34807 (local address 127.0.97.2:33291)
0619 01:09:40.303162 (+ 17367us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.303485 (+ 323us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:40.303742 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:40.304264 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:40.304268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:40.304294 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:40.304658 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.304671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.306124 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.306129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.308415 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.308444 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.310417 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.310427 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.310747 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.310803 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 01:09:40.312180 (+ 1377us) client_negotiation.cc:240] Negotiation successful
0619 01:09:40.312418 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17220,"thread_start_us":17430,"threads_started":1}
I20260619 01:09:40.316944 13704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f161a766bbfc4592bb74ec85a3667962" candidate_uuid: "58c573862fc945908bb2c05d045a0d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629558fb276246329bbb881555c4ade0" is_pre_election: true
I20260619 01:09:40.317687 13704 raft_consensus.cc:2454] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58c573862fc945908bb2c05d045a0d8d in term 0.
I20260619 01:09:40.318595 13503 leader_election.cc:302] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58c573862fc945908bb2c05d045a0d8d, 629558fb276246329bbb881555c4ade0; no voters:
I20260619 01:09:40.319116 13785 raft_consensus.cc:2781] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:09:40.319291 13785 raft_consensus.cc:482] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:09:40.319440 13785 raft_consensus.cc:3037] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:40.322525 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.296627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37811 (local address 127.0.97.1:36483)
0619 01:09:40.296850 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.296856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.315715 (+ 18859us) server_negotiation.cc:407] Connection header received
0619 01:09:40.315998 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.316005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.316092 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.316160 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:40.316980 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.318046 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.320583 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.321483 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.321642 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.321756 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.322146 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.322277 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:09:40.322602 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.296143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36483 (local address 127.0.97.2:37811)
0619 01:09:40.315593 (+ 19450us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.315618 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:40.315835 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:40.316362 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:40.316366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:40.316394 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:40.316778 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.316792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.318242 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.318248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.320403 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:40.320419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.321737 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.321745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:40.321901 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.321926 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 01:09:40.322109 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 01:09:40.322259 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17099,"thread_start_us":16716,"threads_started":1}
I20260619 01:09:40.323056 13785 raft_consensus.cc:504] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629558fb276246329bbb881555c4ade0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34807 } } peers { permanent_uuid: "58c573862fc945908bb2c05d045a0d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36301 } } peers { permanent_uuid: "434a8706fea84a218dde406d5cebc674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36483 } }
I20260619 01:09:40.323841 13785 leader_election.cc:288] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [CANDIDATE]: Term 1 election: Requested vote from peers 629558fb276246329bbb881555c4ade0 (127.0.97.3:34807), 434a8706fea84a218dde406d5cebc674 (127.0.97.1:36483)
I20260619 01:09:40.323772 13425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f161a766bbfc4592bb74ec85a3667962" candidate_uuid: "58c573862fc945908bb2c05d045a0d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "434a8706fea84a218dde406d5cebc674" is_pre_election: true
I20260619 01:09:40.324400 13704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f161a766bbfc4592bb74ec85a3667962" candidate_uuid: "58c573862fc945908bb2c05d045a0d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629558fb276246329bbb881555c4ade0"
I20260619 01:09:40.324406 13425 raft_consensus.cc:2454] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58c573862fc945908bb2c05d045a0d8d in term 0.
I20260619 01:09:40.324635 13704 raft_consensus.cc:3037] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:40.324656 13424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f161a766bbfc4592bb74ec85a3667962" candidate_uuid: "58c573862fc945908bb2c05d045a0d8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "434a8706fea84a218dde406d5cebc674"
I20260619 01:09:40.324908 13424 raft_consensus.cc:3037] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:40.327167 13704 raft_consensus.cc:2454] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58c573862fc945908bb2c05d045a0d8d in term 1.
I20260619 01:09:40.327642 13503 leader_election.cc:302] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58c573862fc945908bb2c05d045a0d8d, 629558fb276246329bbb881555c4ade0; no voters:
I20260619 01:09:40.327941 13785 raft_consensus.cc:2781] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:09:40.328112 13424 raft_consensus.cc:2454] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58c573862fc945908bb2c05d045a0d8d in term 1.
I20260619 01:09:40.328146 13785 raft_consensus.cc:686] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 1 LEADER]: Becoming Leader. State: Replica: 58c573862fc945908bb2c05d045a0d8d, State: Running, Role: LEADER
I20260619 01:09:40.328701 13785 consensus_queue.cc:237] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629558fb276246329bbb881555c4ade0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34807 } } peers { permanent_uuid: "58c573862fc945908bb2c05d045a0d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36301 } } peers { permanent_uuid: "434a8706fea84a218dde406d5cebc674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36483 } }
I20260619 01:09:40.334395 13273 catalog_manager.cc:5495] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 58c573862fc945908bb2c05d045a0d8d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "58c573862fc945908bb2c05d045a0d8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "629558fb276246329bbb881555c4ade0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58c573862fc945908bb2c05d045a0d8d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "434a8706fea84a218dde406d5cebc674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36483 } health_report { overall_health: UNKNOWN } } }
I20260619 01:09:40.381002 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.374706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.0.97.2:36301)
0619 01:09:40.374922 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.374929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.374951 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:09:40.375139 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.375143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.375243 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.375406 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:09:40.375813 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.376910 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.378473 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.379081 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.379211 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.379725 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:09:40.380418 (+ 693us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:09:40.380576 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.380631 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.380719 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:09:40.390822 13424 raft_consensus.cc:1261] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674 [term 1 FOLLOWER]: Refusing update from remote peer 58c573862fc945908bb2c05d045a0d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:09:40.390784 13704 raft_consensus.cc:1261] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0 [term 1 FOLLOWER]: Refusing update from remote peer 58c573862fc945908bb2c05d045a0d8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:09:40.391887 13784 consensus_queue.cc:1035] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "629558fb276246329bbb881555c4ade0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:09:40.392316 13785 consensus_queue.cc:1035] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "434a8706fea84a218dde406d5cebc674" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:09:40.419859 13793 mvcc.cc:205] Tried to move back new op lower bound from 7298381334069190656 to 7298381333831319552. Current Snapshot: MvccSnapshot[applied={T|T < 7298381334069190656}]
I20260619 01:09:40.436688 13273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:09:40.437023 13273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:09:40.441516 13273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01bea5ad3bca4eb49d036006597d8d0d: Sending DeleteTablet for 3 replicas of tablet f161a766bbfc4592bb74ec85a3667962
I20260619 01:09:40.442966 13550 tablet_service.cc:1514] Processing DeleteTablet for tablet f161a766bbfc4592bb74ec85a3667962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:09:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44528
I20260619 01:09:40.443002 13684 tablet_service.cc:1514] Processing DeleteTablet for tablet f161a766bbfc4592bb74ec85a3667962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:09:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37064
I20260619 01:09:40.444003 13405 tablet_service.cc:1514] Processing DeleteTablet for tablet f161a766bbfc4592bb74ec85a3667962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:09:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40170
I20260619 01:09:40.445696 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 13310
I20260619 01:09:40.465250 13800 tablet_replica.cc:330] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d: stopping tablet replica
I20260619 01:09:40.466177 13800 raft_consensus.cc:2229] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:09:40.470019 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.450256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.0.97.1:36483)
0619 01:09:40.450470 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.450477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.450500 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:09:40.450576 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.450584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.450676 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.450832 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:40.454105 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.455388 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.457266 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.458001 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.458182 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.462239 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:40.462263 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:40.462285 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:40.462333 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:40.466722 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.467623 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.467633 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.467637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.467822 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.468217 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.468222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.468225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.468719 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:40.469026 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.469335 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.469605 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 01:09:40.470531 13802 tablet_replica.cc:330] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674: stopping tablet replica
I20260619 01:09:40.471073 13801 tablet_replica.cc:330] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: stopping tablet replica
I20260619 01:09:40.471107 13465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:40.471410 13800 raft_consensus.cc:2258] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:09:40.471961 13801 raft_consensus.cc:2229] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:09:40.474445 13800 ts_tablet_manager.cc:1905] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:09:40.475041 13801 raft_consensus.cc:2258] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:09:40.485491 13801 ts_tablet_manager.cc:1905] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:09:40.492442 13801 ts_tablet_manager.cc:1918] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:09:40.493350 13801 log.cc:1196] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/f161a766bbfc4592bb74ec85a3667962
I20260619 01:09:40.485231 13800 ts_tablet_manager.cc:1918] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:09:40.493767 13800 log.cc:1196] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/f161a766bbfc4592bb74ec85a3667962
I20260619 01:09:40.494837 13800 ts_tablet_manager.cc:1939] T f161a766bbfc4592bb74ec85a3667962 P 58c573862fc945908bb2c05d045a0d8d: Deleting consensus metadata
I20260619 01:09:40.495024 13801 ts_tablet_manager.cc:1939] T f161a766bbfc4592bb74ec85a3667962 P 629558fb276246329bbb881555c4ade0: Deleting consensus metadata
I20260619 01:09:40.497187 13258 catalog_manager.cc:4841] TS 629558fb276246329bbb881555c4ade0 (127.0.97.3:34807): tablet f161a766bbfc4592bb74ec85a3667962 (table test-table [id=91fbe03ce5194afea181920cac1ff141]) successfully deleted
I20260619 01:09:40.498783 13260 catalog_manager.cc:4841] TS 58c573862fc945908bb2c05d045a0d8d (127.0.97.2:36301): tablet f161a766bbfc4592bb74ec85a3667962 (table test-table [id=91fbe03ce5194afea181920cac1ff141]) successfully deleted
I20260619 01:09:40.479216 13802 raft_consensus.cc:2229] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:09:40.645519 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 13310
I20260619 01:09:40.645618 13802 raft_consensus.cc:2258] T f161a766bbfc4592bb74ec85a3667962 P 434a8706fea84a218dde406d5cebc674 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:09:40.663496 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 13474
W20260619 01:09:40.663594 13259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36483 (error 108)
I20260619 01:09:40.665853 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.665273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36483 (local address 127.0.0.1:40196)
0619 01:09:40.665558 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.665680 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
W20260619 01:09:40.666122 13259 catalog_manager.cc:4584] TS 434a8706fea84a218dde406d5cebc674 (127.0.97.1:36483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f161a766bbfc4592bb74ec85a3667962: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36483: connect: Connection refused (error 111)
I20260619 01:09:40.680547 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.664723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.0.97.2:36301)
0619 01:09:40.664935 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.664940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.664969 (+ 29us) server_negotiation.cc:407] Connection header received
0619 01:09:40.665020 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.665024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.665091 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.665216 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:40.666802 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.667634 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.669131 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.669814 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.669984 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.673115 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:40.673137 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:40.673157 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:40.673201 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:40.677093 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.678089 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.678102 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.678108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.678326 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.678791 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.678796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.678799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.679242 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:40.679599 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.679907 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.680180 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 01:09:40.681340 13610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:40.720249 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.719506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36483 (local address 127.0.0.1:40202)
0619 01:09:40.719885 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.720028 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
I20260619 01:09:40.721570 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.721167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36483 (local address 127.0.0.1:40212)
0619 01:09:40.721346 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.721430 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 01:09:40.782351 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.781845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36483 (local address 127.0.0.1:40222)
0619 01:09:40.782082 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.782180 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 01:09:40.783443 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.783110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36483 (local address 127.0.0.1:40234)
0619 01:09:40.783258 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.783334 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 01:09:40.821242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 13474
I20260619 01:09:40.840313 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 13619
I20260619 01:09:40.859443 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.841115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.0.97.3:34807)
0619 01:09:40.841633 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:40.841639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:40.841653 (+ 14us) server_negotiation.cc:407] Connection header received
0619 01:09:40.841706 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:40.841710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:40.841767 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:40.841888 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:40.843235 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.843947 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.845360 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:40.845823 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:40.845963 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:40.853387 (+ 7424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:40.853404 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:40.853421 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:40.853467 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:40.856271 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.857109 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.857118 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.857120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.857269 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:40.857734 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:40.857740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:40.857743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:40.858121 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:40.858450 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:40.858806 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 01:09:40.859046 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":292,"threads_started":1}
I20260619 01:09:40.860173 13744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:40.864259 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.863479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36483 (local address 127.0.0.1:40240)
0619 01:09:40.863825 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.863992 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260619 01:09:40.865584 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:40.865117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36483 (local address 127.0.0.1:40244)
0619 01:09:40.865360 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:40.865457 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260619 01:09:40.952152 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 13619
I20260619 01:09:40.968778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 13240
I20260619 01:09:40.970189 13303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:41.087393 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:41.086766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36483 (local address 127.0.0.1:40250)
0619 01:09:41.087005 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:41.087138 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 01:09:41.087373 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 13240
I20260619 01:09:41.110775 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 01:09:41.457021 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 01:09:41.457329 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 01:09:41.803799 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 01:09:41.804 UTC [12292] LOG: received smart shutdown request
2026-06-19 01:09:41.807 UTC [12292] LOG: background worker "logical replication launcher" (PID 12301) exited with exit code 1
2026-06-19 01:09:41.807 UTC [12296] LOG: shutting down
2026-06-19 01:09:41.963 UTC [12292] LOG: database system is shut down
2026-06-19T01:09:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54733 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:09:42 dist-test-slave-5p8w krb5kdc[13817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:09:42 dist-test-slave-5p8w krb5kdc[13817](info): set up 2 sockets
Jun 19 01:09:42 dist-test-slave-5p8w krb5kdc[13817](info): commencing operation
krb5kdc: starting...
W20260619 01:09:43.189733 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:09:43 dist-test-slave-5p8w krb5kdc[13817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:09:43.428768 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39691
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39691,127.0.97.61:40163,127.0.97.60:44635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:09:43.762290 13830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:43.762622 13830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:43.762718 13830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:43.770619 13830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:09:43.770742 13830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:43.770790 13830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:43.770854 13830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:09:43.770900 13830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:09:43.781497 13830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39691,127.0.97.61:40163,127.0.97.60:44635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:43.782852 13830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:43.784174 13830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:43.796073 13835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:43.796273 13836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:43.796622 13838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:43.797042 13830 server_base.cc:1034] running on GCE node
I20260619 01:09:43.797907 13830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:43.799392 13830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:43.800601 13830 hybrid_clock.cc:648] HybridClock initialized: now 1781831383800540 us; error 64 us; skew 500 ppm
I20260619 01:09:43.800941 13830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:43 dist-test-slave-5p8w krb5kdc[13817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:43.804332 13830 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:09:43.806135 13830 webserver.cc:460] Webserver started at http://127.0.97.62:34649/ using document root <none> and password file <none>
I20260619 01:09:43.806915 13830 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:43.807080 13830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:43.807442 13830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:09:43.810871 13830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "856e9336acde4725871f220b1199705a"
format_stamp: "Formatted at 2026-06-19 01:09:43 on dist-test-slave-5p8w"
server_key: "0aa06dc7413d23f18752401aba128ee3"
server_key_iv: "fd4c6042a707dfac5423708f18328387"
server_key_version: "encryptionkey@0"
I20260619 01:09:43.811784 13830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "856e9336acde4725871f220b1199705a"
format_stamp: "Formatted at 2026-06-19 01:09:43 on dist-test-slave-5p8w"
server_key: "0aa06dc7413d23f18752401aba128ee3"
server_key_iv: "fd4c6042a707dfac5423708f18328387"
server_key_version: "encryptionkey@0"
I20260619 01:09:43.817710 13830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 01:09:43.821617 13845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:43.823233 13830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:09:43.823464 13830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "856e9336acde4725871f220b1199705a"
format_stamp: "Formatted at 2026-06-19 01:09:43 on dist-test-slave-5p8w"
server_key: "0aa06dc7413d23f18752401aba128ee3"
server_key_iv: "fd4c6042a707dfac5423708f18328387"
server_key_version: "encryptionkey@0"
I20260619 01:09:43.823715 13830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:43.839442 13830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:43.844131 13830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:43.844460 13830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:43.863684 13830 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39691
I20260619 01:09:43.863722 13896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39691 every 8 connection(s)
I20260619 01:09:43.865163 13830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:09:43.869269 13897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:43.871920 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 13830
I20260619 01:09:43.872172 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:09:43.872830 388 external_mini_cluster.cc:1428] Setting key 208a47ed6b1709dbad786a309038a4c9
I20260619 01:09:43.877177 13897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 19 01:09:43 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:43.904493 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:43.882599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.0.97.62:39691)
0619 01:09:43.883445 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:43.883460 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:43.883551 (+ 91us) server_negotiation.cc:407] Connection header received
0619 01:09:43.884994 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:43.885004 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:43.885368 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:43.885628 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:43.887161 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:43.888531 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:43.890213 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:43.890876 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:43.891120 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:43.896783 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:43.896792 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:43.896808 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:43.896842 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:43.899839 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:43.900620 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:43.900628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:43.900630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:43.900783 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:43.901180 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:43.901187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:43.901195 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:43.901805 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:43.902022 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:43.902575 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 01:09:43.902937 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":629,"thread_start_us":413,"threads_started":1}
I20260619 01:09:43.904456 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:43.881013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39691 (local address 127.0.0.1:43162)
0619 01:09:43.882009 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:43.882068 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:43.884470 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:43.885648 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:43.885656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:43.886511 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:43.886913 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:43.886934 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:43.888719 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:43.888727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:43.890046 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:43.890061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:43.891167 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:43.891174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:43.891390 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:43.892170 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:43.892197 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:43.896499 (+ 4302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:43.900006 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:43.900020 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:43.900035 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:43.900466 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:43.900894 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:43.900900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:43.900903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:43.901068 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:43.901944 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:43.901952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:43.902181 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 01:09:43.902441 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 01:09:43.903055 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":158,"thread_start_us":433,"threads_started":1}
I20260619 01:09:43.908160 13897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:09:43.909636 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:43.909004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43354)
0619 01:09:43.909248 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:43.909441 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260619 01:09:43.910039 13848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
I20260619 01:09:43.912052 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:43.911470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43366)
0619 01:09:43.911697 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:43.911838 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 01:09:43.912298 13897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40163: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 01:09:43.925380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40163
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39691,127.0.97.61:40163,127.0.97.60:44635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:09:43.961712 13897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } attempt: 1
I20260619 01:09:43.963037 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:43.962501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43380)
0619 01:09:43.962730 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:43.962853 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 01:09:43.964152 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:43.963755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43396)
0619 01:09:43.963914 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:43.963998 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 01:09:43.964370 13897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40163: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
I20260619 01:09:44.032626 13897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } attempt: 2
I20260619 01:09:44.033979 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.033395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43408)
0619 01:09:44.033664 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.033791 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 01:09:44.035097 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.034668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43414)
0619 01:09:44.034828 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.034934 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 01:09:44.035315 13897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40163: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
I20260619 01:09:44.126601 13897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } attempt: 3
I20260619 01:09:44.127888 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.127323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43430)
0619 01:09:44.127553 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.127686 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 01:09:44.129094 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.128553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43440)
0619 01:09:44.128769 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.128874 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260619 01:09:44.129341 13897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40163: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
I20260619 01:09:44.272614 13897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } attempt: 4
I20260619 01:09:44.274001 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.273391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43442)
0619 01:09:44.273714 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.273830 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 01:09:44.275101 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.274675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43454)
0619 01:09:44.274835 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.274939 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 01:09:44.275311 13897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40163: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40163: connect: Connection refused (error 111)
W20260619 01:09:44.278935 13904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:44.279326 13904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:44.279501 13904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:44.287874 13904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:09:44.288003 13904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:44.288050 13904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:44.288108 13904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:09:44.288151 13904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:09:44.298921 13904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39691,127.0.97.61:40163,127.0.97.60:44635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:44.300283 13904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:44.301669 13904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:44.314078 13909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:44.314118 13912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:44.314078 13910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:44.315104 13904 server_base.cc:1034] running on GCE node
I20260619 01:09:44.316092 13904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:44.318063 13904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:44.319391 13904 hybrid_clock.cc:648] HybridClock initialized: now 1781831384319308 us; error 76 us; skew 500 ppm
I20260619 01:09:44.319839 13904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:44 dist-test-slave-5p8w krb5kdc[13817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:44.323352 13904 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 01:09:44.324903 13904 webserver.cc:460] Webserver started at http://127.0.97.61:43653/ using document root <none> and password file <none>
I20260619 01:09:44.325695 13904 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:44.325798 13904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:44.326056 13904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:09:44.329475 13904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/data/instance:
uuid: "45401b03659f4462bc2f66c5a4dffdfa"
format_stamp: "Formatted at 2026-06-19 01:09:44 on dist-test-slave-5p8w"
server_key: "0afd778055f22b4200f988422623488a"
server_key_iv: "49ecc1283c8509dfbfd121c985350c9f"
server_key_version: "encryptionkey@0"
I20260619 01:09:44.330328 13904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/wal/instance:
uuid: "45401b03659f4462bc2f66c5a4dffdfa"
format_stamp: "Formatted at 2026-06-19 01:09:44 on dist-test-slave-5p8w"
server_key: "0afd778055f22b4200f988422623488a"
server_key_iv: "49ecc1283c8509dfbfd121c985350c9f"
server_key_version: "encryptionkey@0"
I20260619 01:09:44.336083 13904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 01:09:44.340091 13919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:44.341775 13904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 01:09:44.341979 13904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/wal
uuid: "45401b03659f4462bc2f66c5a4dffdfa"
format_stamp: "Formatted at 2026-06-19 01:09:44 on dist-test-slave-5p8w"
server_key: "0afd778055f22b4200f988422623488a"
server_key_iv: "49ecc1283c8509dfbfd121c985350c9f"
server_key_version: "encryptionkey@0"
I20260619 01:09:44.342201 13904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:44.363729 13904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:44.369095 13904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:44.369438 13904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:44.388962 13904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40163
I20260619 01:09:44.388988 13970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40163 every 8 connection(s)
I20260619 01:09:44.390498 13904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
I20260619 01:09:44.394759 13971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:44.400233 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 13904
I20260619 01:09:44.400453 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-1/wal/instance
I20260619 01:09:44.400971 388 external_mini_cluster.cc:1428] Setting key 20d75daa7fd801682ad3a2680c0962a0
I20260619 01:09:44.402585 13971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:09:44 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 01:09:44.428849 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.406470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43172 (local address 127.0.97.62:39691)
0619 01:09:44.407045 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.407054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:44.408188 (+ 1134us) server_negotiation.cc:407] Connection header received
0619 01:09:44.410544 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:44.410554 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:44.410662 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:44.410768 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:44.413044 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.413899 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.416676 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.417110 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.417227 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.422015 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:44.422021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:44.422024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:44.422065 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:44.425030 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.426051 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.426057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.426062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.426200 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.426787 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.426791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.426793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.427151 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:44.427320 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:44.428148 (+ 828us) server_negotiation.cc:299] Negotiation successful
0619 01:09:44.428412 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":278,"threads_started":1}
I20260619 01:09:44.430035 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.407111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39691 (local address 127.0.0.1:43172)
0619 01:09:44.407991 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.408043 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:44.410153 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:44.411055 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:44.411069 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:44.411910 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:44.412822 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.412852 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.414085 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.414093 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.416401 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.416421 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.417414 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.417424 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.417808 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.418632 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:44.418669 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:44.421723 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:44.425228 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.425239 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.425275 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.425904 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.426373 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.426378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.426382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.426632 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.427303 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:44.427311 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:44.427752 (+ 441us) client_negotiation.cc:769] Sending connection context
0619 01:09:44.428064 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 01:09:44.428689 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":370,"threads_started":1}
I20260619 01:09:44.432019 13971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:09:44 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:09:44.448916 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.432888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43470)
0619 01:09:44.433082 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.433105 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:44.433230 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:44.434477 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:44.434481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:44.434976 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:44.435106 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.435114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.436479 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.436484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.437426 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.437438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.438306 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.438312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.438519 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.439173 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:44.439191 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:44.441333 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:44.446188 (+ 4855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.446193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.446196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.446618 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.447136 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.447141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.447144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.447337 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.448191 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:44.448195 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:44.448324 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 01:09:44.448429 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:09:44.448641 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 01:09:44.449285 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.433221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.0.97.61:40163)
0619 01:09:44.434031 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.434038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:44.434064 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:09:44.434111 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:44.434118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:44.434322 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:44.434486 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:44.435264 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.436319 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.437559 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.438128 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.438287 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.441614 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:44.441634 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:44.441659 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:44.441712 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:44.445980 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.446781 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.446793 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.446799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.446964 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.447467 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.447474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.447478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.448041 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:44.448277 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:44.448650 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 01:09:44.448898 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":475,"threads_started":1}
I20260619 01:09:44.452062 13971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:09:44.453428 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.452796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45706)
0619 01:09:44.453051 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.453193 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260619 01:09:44.453737 13922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
I20260619 01:09:44.455333 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.454953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45710)
0619 01:09:44.455135 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.455225 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab.
W20260619 01:09:44.455585 13971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
I20260619 01:09:44.456739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44635
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39691,127.0.97.61:40163,127.0.97.60:44635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:09:44.504976 13971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } attempt: 1
I20260619 01:09:44.506381 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.505888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45722)
0619 01:09:44.506100 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.506209 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 01:09:44.507835 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.507304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45728)
0619 01:09:44.507517 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.507646 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 01:09:44.508159 13971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
I20260619 01:09:44.574570 13897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } attempt: 5
I20260619 01:09:44.576390 13971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } attempt: 2
I20260619 01:09:44.577519 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.576985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45732)
0619 01:09:44.577204 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.577335 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 01:09:44.578799 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.578372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45740)
0619 01:09:44.578538 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.578627 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 01:09:44.579115 13971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Jun 19 01:09:44 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:09:44.590502 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.575388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43480)
0619 01:09:44.575621 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.575645 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:44.575797 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:44.576166 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:44.576169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:44.576617 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:44.576905 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.576922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.578398 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.578405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.579527 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.579543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.580530 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.580535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.580702 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.581480 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:44.581509 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:44.583696 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:44.588034 (+ 4338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.588039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.588042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.588461 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.588902 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.588906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.588908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.589041 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.589689 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:44.589692 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:44.589843 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 01:09:44.589960 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 01:09:44.590225 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 01:09:44.590651 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.575596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.0.97.61:40163)
0619 01:09:44.575814 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.575820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:44.575844 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:09:44.575907 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:44.575912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:44.576017 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:44.576133 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:44.577089 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.578220 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.579682 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.580376 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.580512 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.583920 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:44.583927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:44.583932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:44.583973 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:44.587794 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.588605 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.588612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.588616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.588784 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.589148 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.589152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.589155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.589564 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:44.589712 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:44.590092 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 01:09:44.590350 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 01:09:44.591897 13897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:09:44.593082 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.592579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45748)
0619 01:09:44.592786 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.592891 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 01:09:44.594151 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.593761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45756)
0619 01:09:44.593918 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.594023 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 01:09:44.594383 13897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
I20260619 01:09:44.645705 13897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } attempt: 1
I20260619 01:09:44.647064 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.646491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45760)
0619 01:09:44.646747 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.646866 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 01:09:44.648216 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.647815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45762)
0619 01:09:44.647987 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.648079 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 01:09:44.648445 13897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
I20260619 01:09:44.670360 13971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } attempt: 3
I20260619 01:09:44.671604 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.671057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45764)
0619 01:09:44.671296 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.671407 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 01:09:44.672776 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.672292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45778)
0619 01:09:44.672538 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.672627 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260619 01:09:44.673002 13971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
I20260619 01:09:44.716682 13897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } attempt: 2
I20260619 01:09:44.718055 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.717436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45784)
0619 01:09:44.717707 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.717844 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 01:09:44.719190 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.718783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45800)
0619 01:09:44.718945 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.719035 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 01:09:44.719412 13897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
W20260619 01:09:44.807865 13979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:44.808174 13979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:44.808321 13979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:44.815781 13979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:09:44.815918 13979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:44.816002 13979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:44.816104 13979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:09:44.816165 13979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:09:44.816279 13971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } attempt: 4
I20260619 01:09:44.817526 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.816953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45808)
0619 01:09:44.817203 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.817345 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 01:09:44.818542 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.818177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45822)
0619 01:09:44.818338 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.818416 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 01:09:44.818755 13971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
I20260619 01:09:44.825635 13897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } attempt: 3
I20260619 01:09:44.826853 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.826310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45834)
0619 01:09:44.826533 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.826649 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 01:09:44.827853 13979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39691,127.0.97.61:40163,127.0.97.60:44635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:44.828076 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.827626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45850)
0619 01:09:44.827813 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.827921 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 01:09:44.828313 13897 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44635: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44635: connect: Connection refused (error 111)
I20260619 01:09:44.829491 13979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:44.830828 13979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:44.843206 13984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:44.843297 13987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:44.843140 13985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:44.843959 13979 server_base.cc:1034] running on GCE node
I20260619 01:09:44.844830 13979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:44.846451 13979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:44.847693 13979 hybrid_clock.cc:648] HybridClock initialized: now 1781831384847644 us; error 40 us; skew 500 ppm
I20260619 01:09:44.848076 13979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:44 dist-test-slave-5p8w krb5kdc[13817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:44.851470 13979 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 01:09:44.853147 13979 webserver.cc:460] Webserver started at http://127.0.97.60:41657/ using document root <none> and password file <none>
I20260619 01:09:44.853971 13979 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:44.854115 13979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:44.854440 13979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:09:44.857667 13979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/data/instance:
uuid: "e00a702d913c4d18bac5479a139b2c5a"
format_stamp: "Formatted at 2026-06-19 01:09:44 on dist-test-slave-5p8w"
server_key: "572d87d03041443bbab74d4c6b00c1da"
server_key_iv: "ded70b35a80e1dfd3585656392162430"
server_key_version: "encryptionkey@0"
I20260619 01:09:44.858487 13979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/wal/instance:
uuid: "e00a702d913c4d18bac5479a139b2c5a"
format_stamp: "Formatted at 2026-06-19 01:09:44 on dist-test-slave-5p8w"
server_key: "572d87d03041443bbab74d4c6b00c1da"
server_key_iv: "ded70b35a80e1dfd3585656392162430"
server_key_version: "encryptionkey@0"
I20260619 01:09:44.863852 13979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:09:44.867451 13994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:44.868942 13979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:09:44.869115 13979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/wal
uuid: "e00a702d913c4d18bac5479a139b2c5a"
format_stamp: "Formatted at 2026-06-19 01:09:44 on dist-test-slave-5p8w"
server_key: "572d87d03041443bbab74d4c6b00c1da"
server_key_iv: "ded70b35a80e1dfd3585656392162430"
server_key_version: "encryptionkey@0"
I20260619 01:09:44.869370 13979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:44.881871 13979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:44.886081 13979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:44.886399 13979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:44.907078 13979 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44635
I20260619 01:09:44.907090 14045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44635 every 8 connection(s)
I20260619 01:09:44.908721 13979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
I20260619 01:09:44.910140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 13979
I20260619 01:09:44.910386 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781831171286417-388-0/minicluster-data/master-2/wal/instance
I20260619 01:09:44.910936 388 external_mini_cluster.cc:1428] Setting key 7d07adfa1a6b6e11909d6766412aebf0
I20260619 01:09:44.913841 14046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:44.922644 14046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:09:44 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 01:09:44 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:44.937438 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.912960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.0.97.60:44635)
0619 01:09:44.913815 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.913831 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:44.913873 (+ 42us) server_negotiation.cc:407] Connection header received
0619 01:09:44.915513 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:44.915534 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:44.915934 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:44.916271 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:44.918386 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.919912 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.922036 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.922866 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.923050 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.928616 (+ 5566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:44.928627 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:44.928639 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:44.928682 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:44.932472 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.933426 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.933433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.933440 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.933608 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.934073 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.934076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.934078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.934618 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:44.934842 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:44.935461 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 01:09:44.935938 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":429,"threads_started":1}
I20260619 01:09:44.942646 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.925877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.0.97.62:39691)
0619 01:09:44.926100 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.926109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:44.926739 (+ 630us) server_negotiation.cc:407] Connection header received
0619 01:09:44.927047 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:44.927051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:44.927133 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:44.927219 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:44.928892 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.930016 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.931558 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.932266 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.932403 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.936213 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:44.936220 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:44.936227 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:44.936273 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:44.939399 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.940225 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.940233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.940241 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.940406 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.940906 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.940913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.940917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.941371 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:44.941547 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:44.942067 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 01:09:44.942294 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:09:44.942705 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.925848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39691 (local address 127.0.0.1:43174)
0619 01:09:44.926570 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.926630 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:44.926892 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:44.927353 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:44.927360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:44.928280 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:44.928693 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.928704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.930201 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.930205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.931401 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.931416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.932463 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.932468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.932649 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.933277 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:44.933338 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:44.936007 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:44.939545 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.939557 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.939564 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.940099 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.940554 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.940559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.940562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.940736 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.941520 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:44.941528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:44.941715 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 01:09:44.941905 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 01:09:44.942253 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":332,"threads_started":1}
I20260619 01:09:44.944118 14046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:09:44 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 19 01:09:44 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:09:44.958405 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.940860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.0.97.62:39691)
0619 01:09:44.941558 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.941562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:44.941581 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:09:44.941619 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:44.941622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:44.941679 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:44.941795 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:44.943505 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.944549 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.946465 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.946870 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.946963 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.952101 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:44.952109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:44.952114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:44.952157 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:44.955374 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.956311 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.956315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.956317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.956441 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.956836 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.956840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.956843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.957218 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:44.957556 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:44.957814 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 01:09:44.958033 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":318,"threads_started":1}
I20260619 01:09:44.959342 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.944924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.0.97.61:40163)
0619 01:09:44.945088 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.945092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:44.945276 (+ 184us) server_negotiation.cc:407] Connection header received
0619 01:09:44.945441 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:44.945445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:44.945506 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:44.945582 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:44.946458 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.947417 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.948751 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.949531 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.949677 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.953377 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:44.953383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:44.953386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:44.953421 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:44.956412 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.957225 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.957230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.957232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.957446 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.957894 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.957899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.957902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.958231 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:44.958404 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:44.958863 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 01:09:44.959084 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:09:44.959416 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.944942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40163 (local address 127.0.0.1:43486)
0619 01:09:44.945179 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.945197 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:44.945331 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:44.945636 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:44.945641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:44.946107 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:44.946282 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.946293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.947598 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.947605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.948594 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.948606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.949720 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.949725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.949888 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.950515 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:44.950533 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:44.953125 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:44.956554 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.956562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.956568 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.957097 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.957571 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.957578 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.957582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.957771 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.958378 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:44.958383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:44.958602 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 01:09:44.958746 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 01:09:44.959028 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 01:09:44.960562 14046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:09:44 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 01:09:44 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:09:44.977240 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.961290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45870)
0619 01:09:44.961475 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:44.961494 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:44.961597 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:44.961906 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:44.961911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:44.962420 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:44.962596 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.962608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.964004 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.964009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.965477 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:44.965496 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.966519 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.966525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:44.966717 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.967496 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:44.967524 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:44.970383 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:44.974289 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.974299 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.974305 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.974852 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.975317 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:44.975323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:44.975328 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:44.975502 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:44.976235 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:44.976239 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:44.976367 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 01:09:44.976525 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 01:09:44.976887 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 01:09:44.977664 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.961396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.0.97.60:44635)
0619 01:09:44.961527 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.961530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:44.961544 (+ 14us) server_negotiation.cc:407] Connection header received
0619 01:09:44.961712 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:44.961716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:44.961776 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:44.961847 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:44.962733 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.963825 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.965628 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.966305 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.966431 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.970624 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:44.970632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:44.970636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:44.970684 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:44.974097 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.975020 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.975025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.975027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.975182 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.975659 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.975664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.975667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.976109 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:44.976285 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:44.976640 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 01:09:44.977144 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":192,"server-negotiator.queue_time_us":45}
I20260619 01:09:44.979125 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.960333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.0.97.61:40163)
0619 01:09:44.960483 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.960489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:44.960603 (+ 114us) server_negotiation.cc:407] Connection header received
0619 01:09:44.960785 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:44.960792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:44.960862 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:44.960936 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:44.962634 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.963829 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.965721 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.966408 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.966516 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:44.971796 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:44.971806 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:44.971810 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:44.971862 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:44.975421 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.976523 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.976529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.976534 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.976703 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:44.977199 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:44.977208 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:44.977212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:44.977710 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:44.978000 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:44.978490 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 01:09:44.978872 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 01:09:44.983700 14046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a: Bootstrap starting.
I20260619 01:09:44.984344 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:09:44.987273 14046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:44.988512 14046 log.cc:824] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:44.998941 14046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a: No bootstrap required, opened a new log
I20260619 01:09:45.005729 13897 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } attempt: 4
I20260619 01:09:45.013127 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.989744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.0.97.61:40163)
0619 01:09:44.990202 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.990210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:44.990837 (+ 627us) server_negotiation.cc:407] Connection header received
0619 01:09:44.991159 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:44.991168 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:44.991262 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:44.991359 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:44.993894 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.995351 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.998596 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:44.999552 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:44.999695 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:45.004680 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:45.004689 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:45.004693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:45.004744 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:45.009274 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:45.010371 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:45.010380 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:45.010384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:45.010570 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:45.011039 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:45.011044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:45.011048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:45.011468 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:45.011721 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:45.012389 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 01:09:45.012680 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260619 01:09:45.013026 14046 raft_consensus.cc:348] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } }
I20260619 01:09:45.013671 14046 raft_consensus.cc:374] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:45.013864 14046 raft_consensus.cc:729] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e00a702d913c4d18bac5479a139b2c5a, State: Initialized, Role: FOLLOWER
I20260619 01:09:45.014823 14046 consensus_queue.cc:260] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } }
I20260619 01:09:45.022119 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } } }
I20260619 01:09:45.023152 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.991393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.0.97.60:44635)
0619 01:09:44.991600 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.991606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:45.004494 (+ 12888us) server_negotiation.cc:407] Connection header received
0619 01:09:45.005054 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:45.005060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:45.005155 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:45.005241 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:45.007341 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.008628 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.010865 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.011621 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.011718 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:45.015620 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:45.015627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:45.015631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:45.015680 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:45.019534 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:45.020435 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:45.020443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:45.020450 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:45.020670 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:45.021196 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:45.021202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:45.021204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:45.021678 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:45.022017 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:45.022692 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 01:09:45.022890 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 01:09:45.024590 14046 sys_catalog.cc:564] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:09:45.024755 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:09:45.027942 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:44.989522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.0.97.62:39691)
0619 01:09:44.990015 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:44.990025 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:44.999288 (+ 9263us) server_negotiation.cc:407] Connection header received
0619 01:09:44.999862 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:44.999868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:44.999985 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:45.000063 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:45.002871 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.003999 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.013681 (+ 9682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.014375 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.014490 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:45.019856 (+ 5366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:45.019863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:45.019868 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:45.019920 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:45.024464 (+ 4544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:45.025392 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:45.025397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:45.025400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:45.025560 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:45.026063 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:45.026068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:45.026071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:45.026465 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:45.026719 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:45.027318 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 01:09:45.027583 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306}
Jun 19 01:09:45 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:09:45.042724 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:45.006417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45890)
0619 01:09:45.006798 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:45.006819 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:45.006929 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:45.022491 (+ 15562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:45.022497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:45.022966 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:45.023180 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:45.023193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.026613 (+ 3420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.026625 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:45.028481 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:45.028508 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.030258 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.030271 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:45.030521 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:45.031230 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:45.031262 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:45.034278 (+ 3016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:45.038786 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:45.038795 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:45.038801 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:45.039384 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:45.039929 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:45.039937 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:45.039943 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:45.040196 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:45.041214 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:45.041222 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:45.041498 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 01:09:45.041688 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 01:09:45.042141 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260619 01:09:45.044571 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:45.007137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.0.97.60:44635)
0619 01:09:45.021956 (+ 14819us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:45.021963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:45.021991 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:09:45.022042 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:45.022047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:45.022131 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:45.024649 (+ 2518us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:45.025008 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.026431 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.028997 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.030057 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.030255 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:45.034549 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:45.034555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:45.034560 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:45.034604 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:45.038570 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:45.039547 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:45.039553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:45.039557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:45.039741 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:45.040376 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:45.040384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:45.040390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:45.040939 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:45.041118 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:45.043501 (+ 2383us) server_negotiation.cc:299] Negotiation successful
0619 01:09:45.043858 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14647,"thread_start_us":15091,"threads_started":1}
I20260619 01:09:45.039326 13979 master_runner.cc:186] Error getting master registration for 127.0.97.62:39691: OK, instance_id { permanent_uuid: "856e9336acde4725871f220b1199705a" instance_seqno: 1781831383858263 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:09:45.048528 13979 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 01:09:45.051255 13897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a: Bootstrap starting.
W20260619 01:09:45.052119 14072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:09:45.052246 14072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:09:45.054809 13897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:45.055892 13897 log.cc:824] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:45.059136 13897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a: No bootstrap required, opened a new log
I20260619 01:09:45.067457 13897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } }
I20260619 01:09:45.067797 13897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:45.067945 13897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856e9336acde4725871f220b1199705a, State: Initialized, Role: FOLLOWER
I20260619 01:09:45.068548 13897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } }
I20260619 01:09:45.069504 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } } }
I20260619 01:09:45.069963 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:09:45.070674 13897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [sys.catalog]: configured and running, proceeding with master startup.
W20260619 01:09:45.082669 14086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:09:45.082917 14086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:09:45.082146 13830 master_runner.cc:186] Error getting master registration for 127.0.97.61:40163: OK, instance_id { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" instance_seqno: 1781831384383217 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:09:45.093958 13830 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 01:09:45.097604 13851 tablet.cc:2369] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:09:45.117990 13971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } attempt: 5
Jun 19 01:09:45 dist-test-slave-5p8w krb5kdc[13817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:09:45.136538 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:45.118785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44635 (local address 127.0.0.1:45898)
0619 01:09:45.119082 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:45.119105 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:45.119267 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:45.119925 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:45.119931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:45.120513 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:45.120730 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:45.120746 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.122389 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.122395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:45.123842 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:45.123859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.124898 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.124905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:45.125113 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:45.125858 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:45.125882 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:45.128907 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:45.133496 (+ 4589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:45.133501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:45.133506 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:45.134066 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:45.134554 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:45.134559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:45.134563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:45.134727 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:45.135468 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:45.135473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:45.135646 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 01:09:45.135804 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 01:09:45.136127 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260619 01:09:45.137710 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:45.119065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.0.97.60:44635)
0619 01:09:45.119331 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:45.119340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:45.119367 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:09:45.119450 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:45.119456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:45.119560 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:45.119691 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:45.120964 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.122081 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.124007 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:45.124732 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:45.124867 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:45.129216 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:45.129223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:45.129229 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:45.129271 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:45.133285 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:45.134241 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:45.134247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:45.134250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:45.134415 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:45.134865 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:45.134870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:45.134873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:45.135258 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:45.135405 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:45.137014 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0619 01:09:45.137308 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 01:09:45.139326 14000 tablet.cc:2369] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:09:45.144127 13971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa: Bootstrap starting.
I20260619 01:09:45.148566 13971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:45.150033 13971 log.cc:824] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:45.153590 13971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa: No bootstrap required, opened a new log
I20260619 01:09:45.165257 13971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } }
I20260619 01:09:45.165752 13971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:45.165881 13971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45401b03659f4462bc2f66c5a4dffdfa, State: Initialized, Role: FOLLOWER
I20260619 01:09:45.166604 13971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } }
I20260619 01:09:45.167642 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } } }
I20260619 01:09:45.168119 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:09:45.169090 13971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:09:45.191712 13904 master_runner.cc:263] No leader master found from master 45401b03659f4462bc2f66c5a4dffdfa
I20260619 01:09:45.191892 13904 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 01:09:45.197937 14100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:09:45.198127 14100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:09:45.213820 14061 raft_consensus.cc:482] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:09:45.214214 14061 raft_consensus.cc:504] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } }
I20260619 01:09:45.216006 14061 leader_election.cc:288] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 856e9336acde4725871f220b1199705a (127.0.97.62:39691), 45401b03659f4462bc2f66c5a4dffdfa (127.0.97.61:40163)
I20260619 01:09:45.217931 13872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e00a702d913c4d18bac5479a139b2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856e9336acde4725871f220b1199705a" is_pre_election: true
I20260619 01:09:45.218592 13872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e00a702d913c4d18bac5479a139b2c5a in term 0.
I20260619 01:09:45.219465 13996 leader_election.cc:302] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856e9336acde4725871f220b1199705a, e00a702d913c4d18bac5479a139b2c5a; no voters:
I20260619 01:09:45.220657 13946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e00a702d913c4d18bac5479a139b2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45401b03659f4462bc2f66c5a4dffdfa" is_pre_election: true
I20260619 01:09:45.221446 13946 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e00a702d913c4d18bac5479a139b2c5a in term 0.
I20260619 01:09:45.225394 14061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:09:45.225594 14061 raft_consensus.cc:482] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:09:45.225703 14061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:45.229682 14061 raft_consensus.cc:504] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } }
I20260619 01:09:45.231115 13872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e00a702d913c4d18bac5479a139b2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856e9336acde4725871f220b1199705a"
I20260619 01:09:45.231379 13872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:45.235057 13872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e00a702d913c4d18bac5479a139b2c5a in term 1.
I20260619 01:09:45.235670 13996 leader_election.cc:302] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856e9336acde4725871f220b1199705a, e00a702d913c4d18bac5479a139b2c5a; no voters:
I20260619 01:09:45.239678 13946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e00a702d913c4d18bac5479a139b2c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45401b03659f4462bc2f66c5a4dffdfa"
I20260619 01:09:45.239948 13946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:45.243999 13946 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e00a702d913c4d18bac5479a139b2c5a in term 1.
I20260619 01:09:45.249622 14061 leader_election.cc:288] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [CANDIDATE]: Term 1 election: Requested vote from peers 856e9336acde4725871f220b1199705a (127.0.97.62:39691), 45401b03659f4462bc2f66c5a4dffdfa (127.0.97.61:40163)
I20260619 01:09:45.250173 14061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:09:45.250662 14061 raft_consensus.cc:686] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [term 1 LEADER]: Becoming Leader. State: Replica: e00a702d913c4d18bac5479a139b2c5a, State: Running, Role: LEADER
I20260619 01:09:45.251502 14061 consensus_queue.cc:237] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } }
I20260619 01:09:45.264674 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e00a702d913c4d18bac5479a139b2c5a. Latest consensus state: current_term: 1 leader_uuid: "e00a702d913c4d18bac5479a139b2c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } } }
I20260619 01:09:45.265314 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:45.276849 14104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:09:45.279398 14104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:09:45.300326 13946 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [term 1 FOLLOWER]: Refusing update from remote peer e00a702d913c4d18bac5479a139b2c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:09:45.301580 14102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:09:45.309227 13872 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [term 1 FOLLOWER]: Refusing update from remote peer e00a702d913c4d18bac5479a139b2c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:09:45.310418 14102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:09:45.348631 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e00a702d913c4d18bac5479a139b2c5a. Latest consensus state: current_term: 1 leader_uuid: "e00a702d913c4d18bac5479a139b2c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } } }
I20260619 01:09:45.348955 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:09:45.352990 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e00a702d913c4d18bac5479a139b2c5a. Latest consensus state: current_term: 1 leader_uuid: "e00a702d913c4d18bac5479a139b2c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } } }
I20260619 01:09:45.353443 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:09:45.357864 14105 mvcc.cc:205] Tried to move back new op lower bound from 7298381354176847872 to 7298381354002386944. Current Snapshot: MvccSnapshot[applied={T|T < 7298381354176847872}]
I20260619 01:09:45.362898 14107 mvcc.cc:205] Tried to move back new op lower bound from 7298381354176847872 to 7298381354002386944. Current Snapshot: MvccSnapshot[applied={T|T < 7298381354176847872}]
I20260619 01:09:45.366544 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e00a702d913c4d18bac5479a139b2c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } } }
I20260619 01:09:45.366854 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:09:45.369455 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e00a702d913c4d18bac5479a139b2c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } } }
I20260619 01:09:45.369769 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:45.370107 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e00a702d913c4d18bac5479a139b2c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } } }
I20260619 01:09:45.370383 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:45.370709 14106 mvcc.cc:205] Tried to move back new op lower bound from 7298381354176847872 to 7298381354002386944. Current Snapshot: MvccSnapshot[applied={T|T < 7298381354176847872}]
I20260619 01:09:45.371954 14104 catalog_manager.cc:1347] Generated new cluster ID: 5b0eb82695f340b4beebe380528add5c
I20260619 01:09:45.372045 14104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:09:45.385974 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e00a702d913c4d18bac5479a139b2c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856e9336acde4725871f220b1199705a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39691 } } peers { permanent_uuid: "45401b03659f4462bc2f66c5a4dffdfa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40163 } } peers { permanent_uuid: "e00a702d913c4d18bac5479a139b2c5a" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44635 } } }
I20260619 01:09:45.386296 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:09:45.415218 14104 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:09:45.417316 14104 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:09:45.439193 14104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e00a702d913c4d18bac5479a139b2c5a: Generated new TSK 0
I20260619 01:09:45.440479 14104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:09:46.084934 14086 catalog_manager.cc:1259] Loaded cluster ID: 5b0eb82695f340b4beebe380528add5c
I20260619 01:09:46.085044 14086 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a: loading cluster ID for follower catalog manager: success
I20260619 01:09:46.087690 14086 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a: acquiring CA information for follower catalog manager: success
I20260619 01:09:46.089722 14086 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 856e9336acde4725871f220b1199705a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 01:09:46.200088 14100 catalog_manager.cc:1259] Loaded cluster ID: 5b0eb82695f340b4beebe380528add5c
I20260619 01:09:46.200197 14100 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa: loading cluster ID for follower catalog manager: success
I20260619 01:09:46.202760 14100 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa: acquiring CA information for follower catalog manager: success
I20260619 01:09:46.204679 14100 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45401b03659f4462bc2f66c5a4dffdfa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 01:09:46.584263 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 13830
I20260619 01:09:46.585670 13892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:46.697767 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 13830
I20260619 01:09:46.720849 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 13904
I20260619 01:09:46.722148 13966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:46.829957 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 13904
I20260619 01:09:46.850222 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 13979
I20260619 01:09:46.851413 14041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:46.960949 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 13979
2026-06-19T01:09:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4873 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:09:47 dist-test-slave-5p8w krb5kdc[14122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:09:47 dist-test-slave-5p8w krb5kdc[14122](info): set up 2 sockets
Jun 19 01:09:47 dist-test-slave-5p8w krb5kdc[14122](info): commencing operation
krb5kdc: starting...
W20260619 01:09:48.077826 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:09:48 dist-test-slave-5p8w krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:09:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:09:48.315084 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43071
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:09:48.654490 14135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:48.654793 14135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:48.654958 14135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:48.662256 14135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:09:48.662387 14135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:48.662469 14135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:48.662557 14135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:09:48.662639 14135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:09:48.673161 14135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:48.674584 14135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:48.675868 14135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:48.687973 14140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:48.687996 14141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:48.688244 14143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:48.688731 14135 server_base.cc:1034] running on GCE node
I20260619 01:09:48.689659 14135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:48.691290 14135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:48.692535 14135 hybrid_clock.cc:648] HybridClock initialized: now 1781831388692483 us; error 45 us; skew 500 ppm
I20260619 01:09:48.692906 14135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:48 dist-test-slave-5p8w krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:48.696276 14135 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:09:48.697935 14135 webserver.cc:460] Webserver started at http://127.0.97.62:46521/ using document root <none> and password file <none>
I20260619 01:09:48.698657 14135 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:48.698802 14135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:48.699204 14135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:09:48.702582 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "6a5318b8fb6d41c28bdb3c6db6b1c3ae"
format_stamp: "Formatted at 2026-06-19 01:09:48 on dist-test-slave-5p8w"
server_key: "644ec3c0043c56537d5bc6037f81cec1"
server_key_iv: "988cac3073b7ac66e6882354b34d9b13"
server_key_version: "encryptionkey@0"
I20260619 01:09:48.703433 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "6a5318b8fb6d41c28bdb3c6db6b1c3ae"
format_stamp: "Formatted at 2026-06-19 01:09:48 on dist-test-slave-5p8w"
server_key: "644ec3c0043c56537d5bc6037f81cec1"
server_key_iv: "988cac3073b7ac66e6882354b34d9b13"
server_key_version: "encryptionkey@0"
I20260619 01:09:48.709626 14135 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 01:09:48.713265 14150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:48.714775 14135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:09:48.714998 14135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "6a5318b8fb6d41c28bdb3c6db6b1c3ae"
format_stamp: "Formatted at 2026-06-19 01:09:48 on dist-test-slave-5p8w"
server_key: "644ec3c0043c56537d5bc6037f81cec1"
server_key_iv: "988cac3073b7ac66e6882354b34d9b13"
server_key_version: "encryptionkey@0"
I20260619 01:09:48.715216 14135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:48.739298 14135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:48.744043 14135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:48.744350 14135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:48.763260 14135 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43071
I20260619 01:09:48.763295 14201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43071 every 8 connection(s)
I20260619 01:09:48.764704 14135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:09:48.767915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 14135
I20260619 01:09:48.768157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:09:48.768666 14202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:48.768870 388 external_mini_cluster.cc:1428] Setting key 4e64e9ea2e167c795771ec2955abe4eb
I20260619 01:09:48.784302 14202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae: Bootstrap starting.
Jun 19 01:09:48 dist-test-slave-5p8w krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:48.787773 14202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:48.788916 14202 log.cc:824] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:48.792039 14202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae: No bootstrap required, opened a new log
I20260619 01:09:48.793938 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:48.770765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.0.97.62:43071)
0619 01:09:48.771568 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:48.771580 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:48.771625 (+ 45us) server_negotiation.cc:407] Connection header received
0619 01:09:48.773656 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:48.773678 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:48.774118 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:48.774462 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:48.776436 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:48.777848 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:48.779782 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:48.780508 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:48.780651 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:48.786061 (+ 5410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:48.786069 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:48.786085 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:48.786133 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:48.789439 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:48.790243 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:48.790255 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:48.790265 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:48.790524 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:48.790947 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:48.790954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:48.790958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:48.791517 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:48.791840 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:48.792317 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 01:09:48.792790 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":369,"threads_started":1}
I20260619 01:09:48.798303 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:09:48.800843 14202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5318b8fb6d41c28bdb3c6db6b1c3ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43071 } }
I20260619 01:09:48.801343 14202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:48.801474 14202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a5318b8fb6d41c28bdb3c6db6b1c3ae, State: Initialized, Role: FOLLOWER
I20260619 01:09:48.802102 14202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5318b8fb6d41c28bdb3c6db6b1c3ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43071 } }
I20260619 01:09:48.802352 14202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:09:48.802500 14202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:09:48.802680 14202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:48.804929 14202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5318b8fb6d41c28bdb3c6db6b1c3ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43071 } }
I20260619 01:09:48.805559 14202 leader_election.cc:302] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a5318b8fb6d41c28bdb3c6db6b1c3ae; no voters:
I20260619 01:09:48.806161 14202 leader_election.cc:288] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:09:48.806311 14207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:09:48.807056 14207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [term 1 LEADER]: Becoming Leader. State: Replica: 6a5318b8fb6d41c28bdb3c6db6b1c3ae, State: Running, Role: LEADER
I20260619 01:09:48.807648 14207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5318b8fb6d41c28bdb3c6db6b1c3ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43071 } }
I20260619 01:09:48.808429 14202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:09:48.810858 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a5318b8fb6d41c28bdb3c6db6b1c3ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5318b8fb6d41c28bdb3c6db6b1c3ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43071 } } }
I20260619 01:09:48.810823 14209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a5318b8fb6d41c28bdb3c6db6b1c3ae. Latest consensus state: current_term: 1 leader_uuid: "6a5318b8fb6d41c28bdb3c6db6b1c3ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a5318b8fb6d41c28bdb3c6db6b1c3ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43071 } } }
I20260619 01:09:48.811286 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:48.811323 14209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:48.819442 14215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:09:48.825241 14215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:09:48.834841 14215 catalog_manager.cc:1347] Generated new cluster ID: b7e242bf16a0453b8d2e55e573d8f8ba
I20260619 01:09:48.834978 14215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:09:48.849429 14215 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:09:48.850916 14215 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:09:48.861614 14215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a5318b8fb6d41c28bdb3c6db6b1c3ae: Generated new TSK 0
I20260619 01:09:48.862545 14215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:09:48.910902 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 14135
I20260619 01:09:48.912153 14197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:49.008958 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 14135
2026-06-19T01:09:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2043 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:09:49 dist-test-slave-5p8w krb5kdc[14230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:09:49 dist-test-slave-5p8w krb5kdc[14230](info): set up 2 sockets
Jun 19 01:09:49 dist-test-slave-5p8w krb5kdc[14230](info): commencing operation
krb5kdc: starting...
W20260619 01:09:50.101991 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:09:50 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:09:50.350829 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41331
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:09:50.686322 14243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:50.686635 14243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:50.686724 14243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:50.694216 14243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:09:50.694336 14243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:50.694384 14243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:50.694442 14243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:09:50.694479 14243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:09:50.704985 14243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:50.706440 14243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:50.707763 14243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:50.720463 14248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:50.720553 14251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:50.720413 14249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:50.721459 14243 server_base.cc:1034] running on GCE node
I20260619 01:09:50.722322 14243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:50.723927 14243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:50.725245 14243 hybrid_clock.cc:648] HybridClock initialized: now 1781831390725188 us; error 48 us; skew 500 ppm
I20260619 01:09:50.725683 14243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:50 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:50.734071 14243 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:09:50.735879 14243 webserver.cc:460] Webserver started at http://127.0.97.62:34357/ using document root <none> and password file <none>
I20260619 01:09:50.736657 14243 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:50.736949 14243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:50.737383 14243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:09:50.740724 14243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "3828e0ed77c54d37ae263f79749ec113"
format_stamp: "Formatted at 2026-06-19 01:09:50 on dist-test-slave-5p8w"
server_key: "936bfed2ab5c9fb58ded52da8474801e"
server_key_iv: "d597f8bd717aa4ca0e1f108196020688"
server_key_version: "encryptionkey@0"
I20260619 01:09:50.741628 14243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "3828e0ed77c54d37ae263f79749ec113"
format_stamp: "Formatted at 2026-06-19 01:09:50 on dist-test-slave-5p8w"
server_key: "936bfed2ab5c9fb58ded52da8474801e"
server_key_iv: "d597f8bd717aa4ca0e1f108196020688"
server_key_version: "encryptionkey@0"
I20260619 01:09:50.747470 14243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 01:09:50.751366 14258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:50.752926 14243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:09:50.753158 14243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "3828e0ed77c54d37ae263f79749ec113"
format_stamp: "Formatted at 2026-06-19 01:09:50 on dist-test-slave-5p8w"
server_key: "936bfed2ab5c9fb58ded52da8474801e"
server_key_iv: "d597f8bd717aa4ca0e1f108196020688"
server_key_version: "encryptionkey@0"
I20260619 01:09:50.753448 14243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:50.791332 14243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:50.797783 14243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:50.798115 14243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:50.818900 14243 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41331
I20260619 01:09:50.818921 14309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41331 every 8 connection(s)
I20260619 01:09:50.820349 14243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:09:50.824245 14310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:50.825181 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 14243
I20260619 01:09:50.825436 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:09:50.826110 388 external_mini_cluster.cc:1428] Setting key b941d4f88176b59fa7c778f0ae5eaa34
I20260619 01:09:50.839176 14310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Bootstrap starting.
Jun 19 01:09:50 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:50.842897 14310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:50.844019 14310 log.cc:824] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:50.847592 14310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: No bootstrap required, opened a new log
I20260619 01:09:50.850909 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:50.828176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.0.97.62:41331)
0619 01:09:50.829045 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:50.829062 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:50.829107 (+ 45us) server_negotiation.cc:407] Connection header received
0619 01:09:50.830801 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:50.830825 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:50.831259 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:50.831620 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:50.833464 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:50.834792 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:50.836746 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:50.837541 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:50.837683 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:50.842415 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:50.842423 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:50.842438 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:50.842481 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:50.846163 (+ 3682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:50.846852 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:50.846859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:50.846869 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:50.847088 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:50.847455 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:50.847460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:50.847463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:50.848215 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:50.848566 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:50.848848 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 01:09:50.849441 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":645,"thread_start_us":435,"threads_started":1}
I20260619 01:09:50.856294 14310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:50.856649 14310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:50.856720 14310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3828e0ed77c54d37ae263f79749ec113, State: Initialized, Role: FOLLOWER
I20260619 01:09:50.857347 14310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:50.857577 14310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:09:50.857673 14310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:09:50.857793 14310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:50.860226 14310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:50.860730 14310 leader_election.cc:302] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3828e0ed77c54d37ae263f79749ec113; no voters:
I20260619 01:09:50.861440 14310 leader_election.cc:288] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:09:50.861644 14315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:09:50.862329 14315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 1 LEADER]: Becoming Leader. State: Replica: 3828e0ed77c54d37ae263f79749ec113, State: Running, Role: LEADER
I20260619 01:09:50.862910 14315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:50.863719 14310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:09:50.866174 14316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3828e0ed77c54d37ae263f79749ec113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } } }
I20260619 01:09:50.866115 14317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3828e0ed77c54d37ae263f79749ec113. Latest consensus state: current_term: 1 leader_uuid: "3828e0ed77c54d37ae263f79749ec113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } } }
I20260619 01:09:50.866528 14316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:50.866575 14317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:50.868896 14322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:09:50.874115 14322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:09:50.883020 14322 catalog_manager.cc:1347] Generated new cluster ID: 05f35f0f09814adc82b44fdd58972148
I20260619 01:09:50.883144 14322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:09:50.894043 14322 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:09:50.895394 14322 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:09:50.902899 14322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Generated new TSK 0
I20260619 01:09:50.903759 14322 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:09:50.957122 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41331
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:09:51.315274 14336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:51.315584 14336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:51.315770 14336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:51.323436 14336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:51.323575 14336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:51.323773 14336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:09:51.334054 14336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:51.335470 14336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:51.336906 14336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:51.351044 14342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:51.351243 14344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:51.351461 14341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:51.352561 14336 server_base.cc:1034] running on GCE node
I20260619 01:09:51.353566 14336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:51.355067 14336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:51.356336 14336 hybrid_clock.cc:648] HybridClock initialized: now 1781831391356284 us; error 63 us; skew 500 ppm
I20260619 01:09:51.356663 14336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:51 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:51.360323 14336 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:09:51.362062 14336 webserver.cc:460] Webserver started at http://127.0.97.1:40731/ using document root <none> and password file <none>
I20260619 01:09:51.362799 14336 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:51.362913 14336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:51.363174 14336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:09:51.366706 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "92a5caac1f564823b07ee4f7c9423544"
format_stamp: "Formatted at 2026-06-19 01:09:51 on dist-test-slave-5p8w"
server_key: "79e820ce1e371ae4161143c6718bce63"
server_key_iv: "bea7adee200fe63b36e54314692c5a32"
server_key_version: "encryptionkey@0"
I20260619 01:09:51.367566 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "92a5caac1f564823b07ee4f7c9423544"
format_stamp: "Formatted at 2026-06-19 01:09:51 on dist-test-slave-5p8w"
server_key: "79e820ce1e371ae4161143c6718bce63"
server_key_iv: "bea7adee200fe63b36e54314692c5a32"
server_key_version: "encryptionkey@0"
I20260619 01:09:51.373728 14336 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 01:09:51.377852 14351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:51.379515 14336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:09:51.379683 14336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "92a5caac1f564823b07ee4f7c9423544"
format_stamp: "Formatted at 2026-06-19 01:09:51 on dist-test-slave-5p8w"
server_key: "79e820ce1e371ae4161143c6718bce63"
server_key_iv: "bea7adee200fe63b36e54314692c5a32"
server_key_version: "encryptionkey@0"
I20260619 01:09:51.379878 14336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:51.398520 14336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:51.403712 14336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:51.403981 14336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:51.405061 14336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:51.406652 14336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:51.406733 14336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:51.406824 14336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:51.406869 14336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:51.441604 14336 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38697
I20260619 01:09:51.441782 14463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38697 every 8 connection(s)
I20260619 01:09:51.443035 14336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:09:51.452138 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 14336
I20260619 01:09:51.452389 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:09:51.453418 388 external_mini_cluster.cc:1428] Setting key 53c20ae4341d30ce3c3b69ec5ba1e449
Jun 19 01:09:51 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:51.467280 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:51.445873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48877 (local address 127.0.97.62:41331)
0619 01:09:51.446359 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:51.446364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:51.447382 (+ 1018us) server_negotiation.cc:407] Connection header received
0619 01:09:51.449334 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:51.449340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:51.449444 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:51.449567 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:51.451202 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:51.451981 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:51.454394 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:51.454841 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:51.455010 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:51.459241 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:51.459252 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:51.459260 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:51.459317 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:51.463269 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:51.464473 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:51.464481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:51.464488 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:51.464638 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:51.465239 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:51.465242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:51.465244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:51.465585 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:51.465733 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:51.466786 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0619 01:09:51.467017 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":250,"threads_started":1}
I20260619 01:09:51.468963 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:51.446485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:48877)
0619 01:09:51.447203 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:51.447250 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:51.449000 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:51.449831 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:51.449840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:51.450415 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:51.451017 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:51.451035 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:51.452159 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:51.452163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:51.454203 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:51.454219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:51.455116 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:51.455126 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:51.455396 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:51.456100 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:51.456129 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:51.458930 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:51.463537 (+ 4607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:51.463546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:51.463564 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:51.464304 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:51.464804 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:51.464810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:51.464815 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:51.465111 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:51.465716 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:51.465728 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:51.466249 (+ 521us) client_negotiation.cc:769] Sending connection context
0619 01:09:51.466636 (+ 387us) client_negotiation.cc:240] Negotiation successful
0619 01:09:51.467285 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":272,"threads_started":1}
I20260619 01:09:51.470850 14464 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41331
I20260619 01:09:51.471217 14464 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:51.472050 14464 heartbeater.cc:511] Master 127.0.97.62:41331 requested a full tablet report, sending...
I20260619 01:09:51.474439 14275 ts_manager.cc:194] Registered new tserver with Master: 92a5caac1f564823b07ee4f7c9423544 (127.0.97.1:38697)
I20260619 01:09:51.476002 14275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48877
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:09:51.513110 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41331
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:09:51.852428 14470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:51.852751 14470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:51.852859 14470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:51.860702 14470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:51.860842 14470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:51.860967 14470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:09:51.871740 14470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:51.873112 14470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:51.874593 14470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:51.889688 14475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:51.889724 14478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:51.889767 14476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:51.895093 14470 server_base.cc:1034] running on GCE node
I20260619 01:09:51.896008 14470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:51.897107 14470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:51.898353 14470 hybrid_clock.cc:648] HybridClock initialized: now 1781831391898313 us; error 69 us; skew 500 ppm
I20260619 01:09:51.898763 14470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:51 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:51.902422 14470 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:09:51.904322 14470 webserver.cc:460] Webserver started at http://127.0.97.2:46095/ using document root <none> and password file <none>
I20260619 01:09:51.905166 14470 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:51.905357 14470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:51.905727 14470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:09:51.909161 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "4ce5e73e194143a7bbada5b3f08dcd15"
format_stamp: "Formatted at 2026-06-19 01:09:51 on dist-test-slave-5p8w"
server_key: "418ff0fc1bfa629f520aa3a7bcb69de6"
server_key_iv: "84546008f258ddb79cb8eeb64e99ee36"
server_key_version: "encryptionkey@0"
I20260619 01:09:51.910238 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4ce5e73e194143a7bbada5b3f08dcd15"
format_stamp: "Formatted at 2026-06-19 01:09:51 on dist-test-slave-5p8w"
server_key: "418ff0fc1bfa629f520aa3a7bcb69de6"
server_key_iv: "84546008f258ddb79cb8eeb64e99ee36"
server_key_version: "encryptionkey@0"
I20260619 01:09:51.916822 14470 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 01:09:51.921043 14485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:51.922653 14470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:09:51.922860 14470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "4ce5e73e194143a7bbada5b3f08dcd15"
format_stamp: "Formatted at 2026-06-19 01:09:51 on dist-test-slave-5p8w"
server_key: "418ff0fc1bfa629f520aa3a7bcb69de6"
server_key_iv: "84546008f258ddb79cb8eeb64e99ee36"
server_key_version: "encryptionkey@0"
I20260619 01:09:51.923101 14470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:51.943475 14470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:51.948977 14470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:51.949429 14470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:51.950598 14470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:51.952302 14470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:51.952430 14470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:51.952602 14470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:51.952687 14470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:51.988135 14470 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38193
I20260619 01:09:51.988169 14597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38193 every 8 connection(s)
I20260619 01:09:51.989722 14470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:09:51.997552 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 14470
I20260619 01:09:51.997756 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:09:51.998245 388 external_mini_cluster.cc:1428] Setting key 6ba5dad631d048b57820898d969cb7cc
Jun 19 01:09:52 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:52.012689 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:51.992865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58311 (local address 127.0.97.62:41331)
0619 01:09:51.993442 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:51.993448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:51.994579 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 01:09:51.996527 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:51.996533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:51.996637 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:51.996773 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:51.998744 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:51.999632 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.001548 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.002002 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.002162 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.006140 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:52.006145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:52.006149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:52.006187 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:52.009215 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:52.010139 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:52.010152 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:52.010155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:52.010299 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:52.010789 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:52.010799 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:52.010801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:52.011091 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:52.011280 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:52.012047 (+ 767us) server_negotiation.cc:299] Negotiation successful
0619 01:09:52.012307 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":385,"thread_start_us":253,"threads_started":1}
I20260619 01:09:52.013787 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:51.993594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:58311)
0619 01:09:51.994391 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:51.994437 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:51.996197 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:51.997043 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:51.997053 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:51.997756 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:51.998507 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:51.998529 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:51.999811 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:51.999821 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.001367 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.001381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.002267 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.002274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.002523 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.003317 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:52.003372 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:52.005824 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:52.009448 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:52.009464 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:52.009480 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:52.009973 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:52.010424 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:52.010430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:52.010449 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:52.010651 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:52.011243 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:52.011252 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:52.011646 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 01:09:52.011948 (+ 302us) client_negotiation.cc:240] Negotiation successful
0619 01:09:52.012537 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":307,"threads_started":1}
I20260619 01:09:52.015496 14598 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41331
I20260619 01:09:52.015821 14598 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:52.016598 14598 heartbeater.cc:511] Master 127.0.97.62:41331 requested a full tablet report, sending...
I20260619 01:09:52.018527 14275 ts_manager.cc:194] Registered new tserver with Master: 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193)
I20260619 01:09:52.019861 14275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58311
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:09:52.054556 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41331
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:09:52.401567 14604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:52.401866 14604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:52.402058 14604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:52.409613 14604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:52.409760 14604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:52.409955 14604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:09:52.419971 14604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:52.421435 14604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:52.422736 14604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:52.437031 14609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:52.436957 14610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:52.437008 14612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:52.442955 14604 server_base.cc:1034] running on GCE node
I20260619 01:09:52.443751 14604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:52.444844 14604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:52.446112 14604 hybrid_clock.cc:648] HybridClock initialized: now 1781831392446060 us; error 46 us; skew 500 ppm
I20260619 01:09:52.446520 14604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:52 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:52.456985 14604 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:09:52.458947 14604 webserver.cc:460] Webserver started at http://127.0.97.3:42191/ using document root <none> and password file <none>
I20260619 01:09:52.459753 14604 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:52.459906 14604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:52.460237 14604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:09:52.463364 14604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "7f0274ee5733401bbd3348400b0dafdc"
format_stamp: "Formatted at 2026-06-19 01:09:52 on dist-test-slave-5p8w"
server_key: "7fdf77b01533e9c2d6ac211d66c15fbd"
server_key_iv: "e2df793f1f348476a6d9b6885a80acec"
server_key_version: "encryptionkey@0"
I20260619 01:09:52.464249 14604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7f0274ee5733401bbd3348400b0dafdc"
format_stamp: "Formatted at 2026-06-19 01:09:52 on dist-test-slave-5p8w"
server_key: "7fdf77b01533e9c2d6ac211d66c15fbd"
server_key_iv: "e2df793f1f348476a6d9b6885a80acec"
server_key_version: "encryptionkey@0"
I20260619 01:09:52.469761 14604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 01:09:52.473408 14619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:52.474812 14604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:09:52.474995 14604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "7f0274ee5733401bbd3348400b0dafdc"
format_stamp: "Formatted at 2026-06-19 01:09:52 on dist-test-slave-5p8w"
server_key: "7fdf77b01533e9c2d6ac211d66c15fbd"
server_key_iv: "e2df793f1f348476a6d9b6885a80acec"
server_key_version: "encryptionkey@0"
I20260619 01:09:52.475224 14604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:52.479480 14464 heartbeater.cc:503] Master 127.0.97.62:41331 was elected leader, sending a full tablet report...
I20260619 01:09:52.493059 14604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:52.497184 14604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:52.497516 14604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:52.498597 14604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:52.499981 14604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:52.500057 14604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:52.500217 14604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:52.500303 14604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:52.534942 14604 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42661
I20260619 01:09:52.535029 14731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42661 every 8 connection(s)
I20260619 01:09:52.536264 14604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:09:52.539736 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 14604
I20260619 01:09:52.539959 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:09:52.540552 388 external_mini_cluster.cc:1428] Setting key 55f55d9a3f19c3e8fc860b374ceb7597
Jun 19 01:09:52 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:52.557644 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.538850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58791 (local address 127.0.97.62:41331)
0619 01:09:52.539389 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:52.539396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:52.540583 (+ 1187us) server_negotiation.cc:407] Connection header received
0619 01:09:52.543112 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:52.543117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:52.543205 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:52.543334 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:52.544988 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.545891 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.547571 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.547997 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.548141 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.551713 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:52.551719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:52.551722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:52.551756 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:52.554452 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:52.555205 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:52.555212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:52.555215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:52.555350 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:52.555788 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:52.555794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:52.555797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:52.556104 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:52.556262 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:52.557003 (+ 741us) server_negotiation.cc:299] Negotiation successful
0619 01:09:52.557232 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":240,"threads_started":1}
I20260619 01:09:52.558656 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.539416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:58791)
0619 01:09:52.540350 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:52.540414 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:52.542844 (+ 2430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:52.543614 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:52.543626 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:52.544221 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:52.544806 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.544824 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.546066 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.546072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.547405 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.547418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.548239 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.548246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.548522 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.549178 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:52.549201 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:52.551474 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:52.554606 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:52.554616 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:52.554636 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:52.555083 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:52.555471 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:52.555476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:52.555479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:52.555673 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:52.556234 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:52.556243 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:52.556613 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 01:09:52.556894 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 01:09:52.557526 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":379,"threads_started":1}
I20260619 01:09:52.560014 14732 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41331
I20260619 01:09:52.560329 14732 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:52.561062 14732 heartbeater.cc:511] Master 127.0.97.62:41331 requested a full tablet report, sending...
I20260619 01:09:52.562511 14275 ts_manager.cc:194] Registered new tserver with Master: 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661)
I20260619 01:09:52.563329 14275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58791
I20260619 01:09:52.568317 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:09:52.585665 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.572880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.0.97.62:41331)
0619 01:09:52.573052 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:52.573057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:52.573617 (+ 560us) server_negotiation.cc:407] Connection header received
0619 01:09:52.573767 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:52.573770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:52.573837 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:52.573937 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:52.575297 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.576027 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.577392 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.577794 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.577886 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.580540 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:52.580545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:52.580548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:52.580581 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:52.583071 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:52.583796 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:52.583802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:52.583806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:52.583935 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:52.584269 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:52.584272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:52.584274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:52.584586 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:52.584767 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:52.585188 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 01:09:52.585433 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:09:52.589630 14275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:09:52.596745 14275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:09:52.636198 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.609543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38193 (local address 127.0.0.1:34216)
0619 01:09:52.619924 (+ 10381us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:52.619984 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:52.620245 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:52.628095 (+ 7850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:52.628107 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:52.628156 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:52.628779 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.628793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.631141 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.631147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.633076 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.633089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.635116 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.635121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.635324 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.635396 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 01:09:52.635534 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 01:09:52.635731 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10011,"mutex_wait_us":24,"thread_start_us":19733,"threads_started":1}
I20260619 01:09:52.637029 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.609740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.0.97.2:38193)
0619 01:09:52.627089 (+ 17349us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:52.627098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:52.627129 (+ 31us) server_negotiation.cc:407] Connection header received
0619 01:09:52.627241 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:52.627249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:52.627567 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:52.629108 (+ 1541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:52.629417 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.630968 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.633241 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.634873 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.635001 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.636446 (+ 1445us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:52.636517 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 01:09:52.636687 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17147,"thread_start_us":17983,"threads_started":1}
I20260619 01:09:52.637818 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.610352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.0.1:48662)
0619 01:09:52.629573 (+ 19221us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:52.629597 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:52.629738 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:52.630522 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:52.630531 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:52.630559 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:52.631256 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.631266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.633544 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.633547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.635095 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.635107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.636997 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.637003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.637173 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.637220 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 01:09:52.637354 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 01:09:52.637492 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19078,"mutex_wait_us":41,"thread_start_us":9005,"threads_started":1}
I20260619 01:09:52.637913 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.611107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.0.97.1:38697)
0619 01:09:52.629180 (+ 18073us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:52.629189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:52.629871 (+ 682us) server_negotiation.cc:407] Connection header received
0619 01:09:52.629995 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:52.630003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:52.630264 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:52.631441 (+ 1177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:52.631716 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.633336 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.635276 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.636793 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.636938 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.637162 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:52.637431 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 01:09:52.637593 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17898,"thread_start_us":17713,"threads_started":1}
I20260619 01:09:52.639204 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.610446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.0.97.3:42661)
0619 01:09:52.630009 (+ 19563us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:52.630017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:52.630049 (+ 32us) server_negotiation.cc:407] Connection header received
0619 01:09:52.630130 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:52.630136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:52.630332 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:52.631388 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:52.631686 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.633336 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.635808 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.637864 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.638322 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.638543 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:52.638787 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 01:09:52.638944 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19356,"thread_start_us":19198,"threads_started":1}
I20260619 01:09:52.639760 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.609732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42661 (local address 127.0.0.1:50352)
0619 01:09:52.623376 (+ 13644us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:52.623396 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:52.623522 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:52.630553 (+ 7031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:52.630557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:52.630575 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:52.631191 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.631210 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.633470 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.633475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.635623 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.635636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.638401 (+ 2765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.638418 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.638602 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.638635 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 01:09:52.638723 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 01:09:52.639473 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13490,"mutex_wait_us":49,"thread_start_us":13330,"threads_started":1}
I20260619 01:09:52.641376 14533 tablet_service.cc:1467] Processing CreateTablet for tablet 83b6a681b7bb4186919a941efbba3452 (DEFAULT_TABLE table=test-table [id=ef1f6ee1b63643538cc3f1b0224f3fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:09:52.641376 14399 tablet_service.cc:1467] Processing CreateTablet for tablet 83b6a681b7bb4186919a941efbba3452 (DEFAULT_TABLE table=test-table [id=ef1f6ee1b63643538cc3f1b0224f3fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:09:52.642928 14399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b6a681b7bb4186919a941efbba3452. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:52.642943 14533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b6a681b7bb4186919a941efbba3452. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:52.643280 14667 tablet_service.cc:1467] Processing CreateTablet for tablet 83b6a681b7bb4186919a941efbba3452 (DEFAULT_TABLE table=test-table [id=ef1f6ee1b63643538cc3f1b0224f3fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:09:52.644446 14667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b6a681b7bb4186919a941efbba3452. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:52.655937 14751 tablet_bootstrap.cc:492] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: Bootstrap starting.
I20260619 01:09:52.657827 14752 tablet_bootstrap.cc:492] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: Bootstrap starting.
I20260619 01:09:52.658912 14753 tablet_bootstrap.cc:492] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: Bootstrap starting.
I20260619 01:09:52.660380 14751 tablet_bootstrap.cc:654] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:52.661744 14752 tablet_bootstrap.cc:654] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:52.661731 14751 log.cc:824] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:52.662431 14753 tablet_bootstrap.cc:654] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:52.662982 14752 log.cc:824] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:52.663919 14753 log.cc:824] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:52.667196 14753 tablet_bootstrap.cc:492] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: No bootstrap required, opened a new log
I20260619 01:09:52.667306 14752 tablet_bootstrap.cc:492] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: No bootstrap required, opened a new log
I20260619 01:09:52.667500 14753 ts_tablet_manager.cc:1397] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 01:09:52.667552 14752 ts_tablet_manager.cc:1397] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260619 01:09:52.669615 14751 tablet_bootstrap.cc:492] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: No bootstrap required, opened a new log
I20260619 01:09:52.669907 14751 ts_tablet_manager.cc:1397] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260619 01:09:52.676038 14752 raft_consensus.cc:348] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } } peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } }
I20260619 01:09:52.676476 14752 raft_consensus.cc:374] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:52.676574 14752 raft_consensus.cc:729] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a5caac1f564823b07ee4f7c9423544, State: Initialized, Role: FOLLOWER
I20260619 01:09:52.677250 14752 consensus_queue.cc:260] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } } peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } }
I20260619 01:09:52.679114 14752 ts_tablet_manager.cc:1428] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260619 01:09:52.679730 14753 raft_consensus.cc:348] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } } peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } }
I20260619 01:09:52.680217 14753 raft_consensus.cc:374] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:52.680328 14753 raft_consensus.cc:729] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce5e73e194143a7bbada5b3f08dcd15, State: Initialized, Role: FOLLOWER
I20260619 01:09:52.680599 14751 raft_consensus.cc:348] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } } peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } }
I20260619 01:09:52.681071 14751 raft_consensus.cc:374] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:52.681201 14751 raft_consensus.cc:729] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f0274ee5733401bbd3348400b0dafdc, State: Initialized, Role: FOLLOWER
I20260619 01:09:52.681047 14753 consensus_queue.cc:260] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } } peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } }
I20260619 01:09:52.681861 14751 consensus_queue.cc:260] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } } peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } }
I20260619 01:09:52.682405 14598 heartbeater.cc:503] Master 127.0.97.62:41331 was elected leader, sending a full tablet report...
I20260619 01:09:52.683095 14753 ts_tablet_manager.cc:1428] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260619 01:09:52.684237 14732 heartbeater.cc:503] Master 127.0.97.62:41331 was elected leader, sending a full tablet report...
I20260619 01:09:52.684921 14751 ts_tablet_manager.cc:1428] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260619 01:09:52.695585 14465 tablet.cc:2369] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:09:52.742031 14599 tablet.cc:2369] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:09:52.788053 14733 tablet.cc:2369] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:09:52.788846 14759 raft_consensus.cc:482] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:09:52.789157 14759 raft_consensus.cc:504] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } } peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } }
I20260619 01:09:52.790946 14759 leader_election.cc:288] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193), 92a5caac1f564823b07ee4f7c9423544 (127.0.97.1:38697)
I20260619 01:09:52.796887 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.791408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40921 (local address 127.0.97.2:38193)
0619 01:09:52.791621 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:52.791627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:52.791652 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:09:52.791814 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:52.791818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:52.791881 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:52.792005 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:52.792439 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.793552 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.795170 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.795956 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.796103 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.796200 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:52.796479 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 01:09:52.796617 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 01:09:52.796936 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.791331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38193 (local address 127.0.97.3:40921)
0619 01:09:52.791517 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:52.791545 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:52.791681 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:52.792039 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:52.792043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:52.792078 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:52.792277 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.792286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.793680 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.793686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.795004 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.795019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.796131 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.796137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.796316 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.796341 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 01:09:52.796452 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 01:09:52.796604 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260619 01:09:52.797868 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.791461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33663 (local address 127.0.97.1:38697)
0619 01:09:52.791641 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:52.791647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:52.792643 (+ 996us) server_negotiation.cc:407] Connection header received
0619 01:09:52.792748 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:52.792754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:52.792828 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:52.792929 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:52.793504 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.794219 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.795984 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.796807 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.796984 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.797087 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:52.797489 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 01:09:52.797622 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:09:52.797979 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.791347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.97.3:33663)
0619 01:09:52.792520 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:52.792538 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:52.792647 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:52.792953 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:52.792956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:52.792977 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:52.793275 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.793282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.794382 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.794402 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.795784 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:52.795796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.797031 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.797037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:52.797248 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.797341 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 01:09:52.797461 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 01:09:52.797597 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":417,"thread_start_us":414,"threads_started":1}
I20260619 01:09:52.797891 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b6a681b7bb4186919a941efbba3452" candidate_uuid: "7f0274ee5733401bbd3348400b0dafdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" is_pre_election: true
I20260619 01:09:52.798494 14553 raft_consensus.cc:2454] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f0274ee5733401bbd3348400b0dafdc in term 0.
I20260619 01:09:52.799053 14419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b6a681b7bb4186919a941efbba3452" candidate_uuid: "7f0274ee5733401bbd3348400b0dafdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a5caac1f564823b07ee4f7c9423544" is_pre_election: true
I20260619 01:09:52.799220 14621 leader_election.cc:302] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ce5e73e194143a7bbada5b3f08dcd15, 7f0274ee5733401bbd3348400b0dafdc; no voters:
I20260619 01:09:52.799660 14759 raft_consensus.cc:2781] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:09:52.799631 14419 raft_consensus.cc:2454] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f0274ee5733401bbd3348400b0dafdc in term 0.
I20260619 01:09:52.799819 14759 raft_consensus.cc:482] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:09:52.799947 14759 raft_consensus.cc:3037] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:52.802506 14759 raft_consensus.cc:504] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } } peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } }
I20260619 01:09:52.803236 14759 leader_election.cc:288] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [CANDIDATE]: Term 1 election: Requested vote from peers 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193), 92a5caac1f564823b07ee4f7c9423544 (127.0.97.1:38697)
I20260619 01:09:52.803869 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b6a681b7bb4186919a941efbba3452" candidate_uuid: "7f0274ee5733401bbd3348400b0dafdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce5e73e194143a7bbada5b3f08dcd15"
I20260619 01:09:52.803896 14419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b6a681b7bb4186919a941efbba3452" candidate_uuid: "7f0274ee5733401bbd3348400b0dafdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a5caac1f564823b07ee4f7c9423544"
I20260619 01:09:52.804133 14419 raft_consensus.cc:3037] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:52.804132 14553 raft_consensus.cc:3037] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:52.807475 14419 raft_consensus.cc:2454] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f0274ee5733401bbd3348400b0dafdc in term 1.
I20260619 01:09:52.807475 14553 raft_consensus.cc:2454] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f0274ee5733401bbd3348400b0dafdc in term 1.
I20260619 01:09:52.808032 14620 leader_election.cc:302] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f0274ee5733401bbd3348400b0dafdc, 92a5caac1f564823b07ee4f7c9423544; no voters:
I20260619 01:09:52.808439 14759 raft_consensus.cc:2781] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:09:52.808914 14759 raft_consensus.cc:686] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 1 LEADER]: Becoming Leader. State: Replica: 7f0274ee5733401bbd3348400b0dafdc, State: Running, Role: LEADER
I20260619 01:09:52.809499 14759 consensus_queue.cc:237] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } } peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } }
I20260619 01:09:52.814770 14273 catalog_manager.cc:5495] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7f0274ee5733401bbd3348400b0dafdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } health_report { overall_health: UNKNOWN } } }
I20260619 01:09:52.873703 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.868052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.0.97.3:42661)
0619 01:09:52.868268 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:52.868273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:52.868292 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:09:52.868510 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:52.868514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:52.868591 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:52.868749 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:09:52.869195 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.870059 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.871515 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.871920 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.872060 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.872567 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:09:52.873088 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:09:52.873268 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:52.873347 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 01:09:52.873431 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 01:09:52.882313 14419 raft_consensus.cc:1261] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Refusing update from remote peer 7f0274ee5733401bbd3348400b0dafdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:09:52.882313 14553 raft_consensus.cc:1261] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 1 FOLLOWER]: Refusing update from remote peer 7f0274ee5733401bbd3348400b0dafdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:09:52.883535 14761 consensus_queue.cc:1035] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:09:52.883957 14759 consensus_queue.cc:1035] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:09:52.898757 14765 mvcc.cc:205] Tried to move back new op lower bound from 7298381385234194432 to 7298381384951508992. Current Snapshot: MvccSnapshot[applied={T|T < 7298381385234194432}]
I20260619 01:09:52.917893 14273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:09:52.918202 14273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:09:52.922346 14273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Sending DeleteTablet for 3 replicas of tablet 83b6a681b7bb4186919a941efbba3452
I20260619 01:09:52.923750 14667 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b6a681b7bb4186919a941efbba3452 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:09:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50352
I20260619 01:09:52.923753 14533 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b6a681b7bb4186919a941efbba3452 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:09:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34216
I20260619 01:09:52.923822 14399 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b6a681b7bb4186919a941efbba3452 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:09:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48662
I20260619 01:09:52.924943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 14336
I20260619 01:09:52.939198 14776 tablet_replica.cc:330] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: stopping tablet replica
I20260619 01:09:52.939736 14776 raft_consensus.cc:2229] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:09:52.940315 14776 raft_consensus.cc:2258] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:09:52.942883 14776 ts_tablet_manager.cc:1905] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:09:52.951181 14776 ts_tablet_manager.cc:1918] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:09:52.951407 14776 log.cc:1196] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/83b6a681b7bb4186919a941efbba3452
I20260619 01:09:52.952010 14776 ts_tablet_manager.cc:1939] T 83b6a681b7bb4186919a941efbba3452 P 92a5caac1f564823b07ee4f7c9423544: Deleting consensus metadata
I20260619 01:09:52.954190 14259 catalog_manager.cc:4841] TS 92a5caac1f564823b07ee4f7c9423544 (127.0.97.1:38697): tablet 83b6a681b7bb4186919a941efbba3452 (table test-table [id=ef1f6ee1b63643538cc3f1b0224f3fb5]) successfully deleted
I20260619 01:09:52.955152 14775 tablet_replica.cc:330] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: stopping tablet replica
I20260619 01:09:52.955808 14775 raft_consensus.cc:2229] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:09:52.956506 14775 raft_consensus.cc:2258] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:09:52.959177 14775 ts_tablet_manager.cc:1905] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:09:52.964681 14774 tablet_replica.cc:330] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: stopping tablet replica
I20260619 01:09:52.965250 14774 raft_consensus.cc:2229] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:09:52.965965 14774 raft_consensus.cc:2258] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:09:52.967023 14775 ts_tablet_manager.cc:1918] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:09:52.967218 14775 log.cc:1196] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/83b6a681b7bb4186919a941efbba3452
I20260619 01:09:52.967780 14775 ts_tablet_manager.cc:1939] T 83b6a681b7bb4186919a941efbba3452 P 4ce5e73e194143a7bbada5b3f08dcd15: Deleting consensus metadata
I20260619 01:09:52.970023 14260 catalog_manager.cc:4841] TS 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193): tablet 83b6a681b7bb4186919a941efbba3452 (table test-table [id=ef1f6ee1b63643538cc3f1b0224f3fb5]) successfully deleted
I20260619 01:09:52.970897 14774 ts_tablet_manager.cc:1905] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 01:09:52 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:09:52.977531 14774 ts_tablet_manager.cc:1918] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:09:52.977748 14774 log.cc:1196] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/83b6a681b7bb4186919a941efbba3452
I20260619 01:09:52.978365 14774 ts_tablet_manager.cc:1939] T 83b6a681b7bb4186919a941efbba3452 P 7f0274ee5733401bbd3348400b0dafdc: Deleting consensus metadata
I20260619 01:09:52.980367 14261 catalog_manager.cc:4841] TS 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661): tablet 83b6a681b7bb4186919a941efbba3452 (table test-table [id=ef1f6ee1b63643538cc3f1b0224f3fb5]) successfully deleted
I20260619 01:09:52.980800 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:52.926433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.0.97.1:38697)
0619 01:09:52.926688 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:52.926697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:52.950284 (+ 23587us) server_negotiation.cc:407] Connection header received
0619 01:09:52.950987 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:52.950996 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:52.951122 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:52.951230 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:52.957196 (+ 5966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.958628 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.961895 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:52.962842 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:52.962985 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:52.972574 (+ 9589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:52.972597 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:52.972617 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:52.972673 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:52.976883 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:52.978012 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:52.978023 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:52.978026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:52.978220 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:52.978793 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:52.978799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:52.978803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:52.979309 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:52.979626 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:52.980091 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 01:09:52.980337 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 01:09:52.981544 14459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:53.076313 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 14336
I20260619 01:09:53.095094 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 14470
Jun 19 01:09:53 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:09:53.113008 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:53.095943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34220 (local address 127.0.97.2:38193)
0619 01:09:53.096140 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:53.096146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:53.096169 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:09:53.096309 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:53.096315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:53.096420 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:53.096531 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:53.097887 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:53.098994 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:53.100320 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:53.100962 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:53.101090 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:53.105712 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:53.105734 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:53.105756 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:53.105805 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:53.109581 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:53.110637 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:53.110648 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:53.110653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:53.110805 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:53.111169 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:53.111175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:53.111179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:53.111666 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:53.111965 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:53.112375 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 01:09:53.112607 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 01:09:53.113783 14593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:53.207810 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 14470
I20260619 01:09:53.225936 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 14604
Jun 19 01:09:53 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:09:53.244731 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:53.226808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.0.97.3:42661)
0619 01:09:53.227047 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:53.227053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:53.227082 (+ 29us) server_negotiation.cc:407] Connection header received
0619 01:09:53.227158 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:53.227164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:53.227258 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:53.227378 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:53.228830 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:53.230065 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:53.231400 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:53.232132 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:53.232295 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:53.237268 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:53.237315 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:53.237338 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:53.237416 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:53.241647 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:53.242439 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:53.242450 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:53.242455 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:53.242647 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:53.243013 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:53.243018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:53.243022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:53.243511 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:53.243876 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:53.244060 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 01:09:53.244297 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 01:09:53.245735 14727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:53.372321 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 14604
I20260619 01:09:53.394069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 14243
I20260619 01:09:53.395181 14305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:53.509121 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 14243
I20260619 01:09:53.531265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41331
--webserver_interface=127.0.97.62
--webserver_port=34357
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:09:53.886708 14782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:53.887019 14782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:53.887207 14782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:53.894601 14782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:09:53.894722 14782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:53.894840 14782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:53.894932 14782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:09:53.895017 14782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:09:53.905210 14782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:53.906754 14782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:53.908136 14782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:53.931751 14791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:53.931829 14788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:53.931871 14789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:53.932662 14782 server_base.cc:1034] running on GCE node
I20260619 01:09:53.933704 14782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:53.935474 14782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:53.936687 14782 hybrid_clock.cc:648] HybridClock initialized: now 1781831393936627 us; error 48 us; skew 500 ppm
I20260619 01:09:53.937111 14782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:53 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:53.940709 14782 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:09:53.942503 14782 webserver.cc:460] Webserver started at http://127.0.97.62:34357/ using document root <none> and password file <none>
I20260619 01:09:53.943352 14782 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:53.943634 14782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:53.950363 14782 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 01:09:53.953516 14798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:53.955169 14782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:09:53.955397 14782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "3828e0ed77c54d37ae263f79749ec113"
format_stamp: "Formatted at 2026-06-19 01:09:50 on dist-test-slave-5p8w"
server_key: "936bfed2ab5c9fb58ded52da8474801e"
server_key_iv: "d597f8bd717aa4ca0e1f108196020688"
server_key_version: "encryptionkey@0"
I20260619 01:09:53.956549 14782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:53.979713 14782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:53.984233 14782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:53.984551 14782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:54.004554 14782 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41331
I20260619 01:09:54.004573 14849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41331 every 8 connection(s)
I20260619 01:09:54.006533 14782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:09:54.014037 14850 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:09:54.015875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 14782
I20260619 01:09:54.016701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38697
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41331
--webserver_port=40731
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:09:54.017774 14850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Bootstrap starting.
I20260619 01:09:54.039875 14850 log.cc:824] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:54.053817 14850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:09:54.054441 14850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Bootstrap complete.
I20260619 01:09:54.064371 14850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:54.065361 14850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3828e0ed77c54d37ae263f79749ec113, State: Initialized, Role: FOLLOWER
I20260619 01:09:54.066042 14850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:54.066318 14850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:09:54.066471 14850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:09:54.066663 14850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:09:54.071592 14850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:54.072201 14850 leader_election.cc:302] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3828e0ed77c54d37ae263f79749ec113; no voters:
I20260619 01:09:54.072957 14850 leader_election.cc:288] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 01:09:54.073103 14854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 01:09:54.075738 14850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:09:54.080601 14854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 2 LEADER]: Becoming Leader. State: Replica: 3828e0ed77c54d37ae263f79749ec113, State: Running, Role: LEADER
I20260619 01:09:54.081393 14854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:54.090478 14855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3828e0ed77c54d37ae263f79749ec113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } } }
I20260619 01:09:54.090991 14855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:54.095682 14856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3828e0ed77c54d37ae263f79749ec113. Latest consensus state: current_term: 2 leader_uuid: "3828e0ed77c54d37ae263f79749ec113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } } }
I20260619 01:09:54.096099 14856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:54.097998 14869 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 01:09:54.099195 14870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:09:54.108168 14870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:09:54.109844 14870 catalog_manager.cc:1259] Loaded cluster ID: 05f35f0f09814adc82b44fdd58972148
I20260619 01:09:54.109961 14870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:09:54.113161 14870 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 01:09:54.131170 14870 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 01:09:54.141654 14870 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:09:54.143553 14870 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Loaded TSK: 0
I20260619 01:09:54.145183 14870 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 01:09:54.385767 14852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:54.386129 14852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:54.386300 14852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:54.394030 14852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:54.394201 14852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:54.394381 14852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:09:54.404803 14852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40731
--tserver_master_addrs=127.0.97.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:54.406358 14852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:54.407897 14852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:54.422729 14878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:54.422782 14877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:54.425742 14880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:54.427496 14852 server_base.cc:1034] running on GCE node
I20260619 01:09:54.428329 14852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:54.429507 14852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:54.430783 14852 hybrid_clock.cc:648] HybridClock initialized: now 1781831394430714 us; error 61 us; skew 500 ppm
I20260619 01:09:54.431170 14852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:54 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:54.434806 14852 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:09:54.436653 14852 webserver.cc:460] Webserver started at http://127.0.97.1:40731/ using document root <none> and password file <none>
I20260619 01:09:54.437505 14852 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:54.437660 14852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:54.443603 14852 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:09:54.446655 14887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:54.448204 14852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:09:54.448390 14852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "92a5caac1f564823b07ee4f7c9423544"
format_stamp: "Formatted at 2026-06-19 01:09:51 on dist-test-slave-5p8w"
server_key: "79e820ce1e371ae4161143c6718bce63"
server_key_iv: "bea7adee200fe63b36e54314692c5a32"
server_key_version: "encryptionkey@0"
I20260619 01:09:54.449467 14852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:54.465060 14852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:54.469882 14852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:54.470183 14852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:54.471369 14852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:54.472807 14852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:54.472898 14852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:54.473018 14852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:54.473097 14852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:54.507083 14852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38697
I20260619 01:09:54.507164 14999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38697 every 8 connection(s)
I20260619 01:09:54.508673 14852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:09:54.511694 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 14852
I20260619 01:09:54.512451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38193
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41331
--webserver_port=46095
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:09:54 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:54.538110 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:54.511983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:51687)
0619 01:09:54.513260 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:54.513362 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:54.516210 (+ 2848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:54.517763 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:54.517776 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:54.518601 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:54.519416 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:54.519444 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:54.521135 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:54.521144 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:54.522484 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:54.522497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:54.524027 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:54.524033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:54.524262 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:54.525080 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:54.525106 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:54.527742 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:54.533170 (+ 5428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:54.533178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:54.533193 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:54.533826 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:54.534466 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:54.534470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:54.534473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:54.534710 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:54.535892 (+ 1182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:54.535900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:54.536254 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 01:09:54.536509 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 01:09:54.536948 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":664,"threads_started":1}
I20260619 01:09:54.538713 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:54.511815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51687 (local address 127.0.97.62:41331)
0619 01:09:54.512548 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:54.512566 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:54.513531 (+ 965us) server_negotiation.cc:407] Connection header received
0619 01:09:54.516801 (+ 3270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:54.516814 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:54.517187 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:54.517563 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:54.519666 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:54.520930 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:54.522635 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:54.523749 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:54.523964 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:54.528101 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:54.528113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:54.528137 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:54.528206 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:54.532926 (+ 4720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:54.534021 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:54.534034 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:54.534045 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:54.534298 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:54.534892 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:54.534900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:54.534905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:54.535683 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:54.535917 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:54.536812 (+ 895us) server_negotiation.cc:299] Negotiation successful
0619 01:09:54.537368 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":296,"threads_started":1}
I20260619 01:09:54.541826 15000 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41331
I20260619 01:09:54.542174 15000 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:54.543043 15000 heartbeater.cc:511] Master 127.0.97.62:41331 requested a full tablet report, sending...
I20260619 01:09:54.545746 14815 ts_manager.cc:194] Registered new tserver with Master: 92a5caac1f564823b07ee4f7c9423544 (127.0.97.1:38697)
I20260619 01:09:54.547262 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51687
W20260619 01:09:54.857960 15004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:54.858264 15004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:54.858451 15004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:54.865971 15004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:54.866123 15004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:54.866314 15004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:09:54.876626 15004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46095
--tserver_master_addrs=127.0.97.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:54.878026 15004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:54.879379 15004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:54.892815 15009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:54.892834 15010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:54.892865 15012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:54.893710 15004 server_base.cc:1034] running on GCE node
I20260619 01:09:54.894384 15004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:54.895375 15004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:54.896601 15004 hybrid_clock.cc:648] HybridClock initialized: now 1781831394896566 us; error 62 us; skew 500 ppm
I20260619 01:09:54.896950 15004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:54 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:54.900100 15004 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:09:54.901602 15004 webserver.cc:460] Webserver started at http://127.0.97.2:46095/ using document root <none> and password file <none>
I20260619 01:09:54.902323 15004 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:54.902483 15004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:54.908214 15004 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:09:54.911230 15019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:54.912609 15004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:09:54.912806 15004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "4ce5e73e194143a7bbada5b3f08dcd15"
format_stamp: "Formatted at 2026-06-19 01:09:51 on dist-test-slave-5p8w"
server_key: "418ff0fc1bfa629f520aa3a7bcb69de6"
server_key_iv: "84546008f258ddb79cb8eeb64e99ee36"
server_key_version: "encryptionkey@0"
I20260619 01:09:54.914031 15004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:54.937139 15004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:54.941877 15004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:54.942209 15004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:54.943257 15004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:54.944693 15004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:54.944823 15004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:54.944967 15004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:54.945057 15004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:54.980655 15004 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38193
I20260619 01:09:54.980742 15131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38193 every 8 connection(s)
I20260619 01:09:54.982223 15004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:09:54.989401 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 15004
I20260619 01:09:54.990031 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42661
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41331
--webserver_port=42191
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:09:54 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:55.003901 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:54.984853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46181 (local address 127.0.97.62:41331)
0619 01:09:54.985073 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:54.985080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:54.986284 (+ 1204us) server_negotiation.cc:407] Connection header received
0619 01:09:54.988057 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:54.988061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:54.988132 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:54.988221 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:54.990040 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:54.991231 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:54.992798 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:54.993610 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:54.993761 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:54.997610 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:54.997617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:54.997621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:54.997659 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:55.000291 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:55.001382 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:55.001386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:55.001389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:55.001526 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:55.002029 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:55.002032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:55.002034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:55.002289 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:55.002410 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:55.003364 (+ 954us) server_negotiation.cc:299] Negotiation successful
0619 01:09:55.003548 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 01:09:55.005496 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:54.985420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:46181)
0619 01:09:54.986118 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:54.986160 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:54.987804 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:54.988438 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:54.988447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:54.989138 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:54.989844 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:54.989863 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:54.991393 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:54.991397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:54.992642 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:54.992653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:54.993825 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:54.993830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:54.994031 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:54.994745 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:54.994773 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:54.997269 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:55.000461 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:55.000474 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:55.000496 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:55.001196 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:55.001641 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:55.001649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:55.001656 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:55.001914 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:55.002412 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:55.002423 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:55.002933 (+ 510us) client_negotiation.cc:769] Sending connection context
0619 01:09:55.003271 (+ 338us) client_negotiation.cc:240] Negotiation successful
0619 01:09:55.003945 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":270,"threads_started":1}
I20260619 01:09:55.007308 15132 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41331
I20260619 01:09:55.007661 15132 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:55.008626 15132 heartbeater.cc:511] Master 127.0.97.62:41331 requested a full tablet report, sending...
I20260619 01:09:55.010562 14815 ts_manager.cc:194] Registered new tserver with Master: 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193)
I20260619 01:09:55.011695 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46181
W20260619 01:09:55.340499 15135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:55.340802 15135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:55.340899 15135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:55.348253 15135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:55.348380 15135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:55.348497 15135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:09:55.358455 15135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42191
--tserver_master_addrs=127.0.97.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:55.359737 15135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:55.361168 15135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:55.374588 15143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:55.374722 15140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:55.374722 15141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:55.375206 15135 server_base.cc:1034] running on GCE node
I20260619 01:09:55.375941 15135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:55.377060 15135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:55.378266 15135 hybrid_clock.cc:648] HybridClock initialized: now 1781831395378187 us; error 76 us; skew 500 ppm
I20260619 01:09:55.378609 15135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:55 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:55.381925 15135 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:09:55.383406 15135 webserver.cc:460] Webserver started at http://127.0.97.3:42191/ using document root <none> and password file <none>
I20260619 01:09:55.384141 15135 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:55.384310 15135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:55.389835 15135 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:09:55.392509 15150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:55.393924 15135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:09:55.394115 15135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "7f0274ee5733401bbd3348400b0dafdc"
format_stamp: "Formatted at 2026-06-19 01:09:52 on dist-test-slave-5p8w"
server_key: "7fdf77b01533e9c2d6ac211d66c15fbd"
server_key_iv: "e2df793f1f348476a6d9b6885a80acec"
server_key_version: "encryptionkey@0"
I20260619 01:09:55.395148 15135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:55.411604 15135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:55.416006 15135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:55.416312 15135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:55.417380 15135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:55.418854 15135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:55.418988 15135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:55.419123 15135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:55.419215 15135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:55.453701 15135 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42661
I20260619 01:09:55.453747 15262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42661 every 8 connection(s)
I20260619 01:09:55.455200 15135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:09:55.457437 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 15135
Jun 19 01:09:55 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:09:55.473086 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.459128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.0.97.62:41331)
0619 01:09:55.459643 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:55.459647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:55.459662 (+ 15us) server_negotiation.cc:407] Connection header received
0619 01:09:55.459753 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:55.459757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:55.459829 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:55.459935 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:55.461637 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.462502 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.463927 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.464395 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.464553 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.467197 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:55.467209 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:55.467213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:55.467242 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:55.470090 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:55.470931 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:55.470938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:55.470942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:55.471082 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:55.471532 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:55.471538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:55.471542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:55.471852 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:55.472109 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:55.472429 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 01:09:55.472680 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":434,"thread_start_us":252,"threads_started":1}
I20260619 01:09:55.476725 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.459129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33763 (local address 127.0.97.62:41331)
0619 01:09:55.459317 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:55.459323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:55.459459 (+ 136us) server_negotiation.cc:407] Connection header received
0619 01:09:55.461717 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:55.461726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:55.461798 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:55.461895 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:55.463352 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.464074 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.465911 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.466352 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.466488 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.470389 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:55.470395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:55.470398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:55.470428 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:55.473376 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:55.474224 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:55.474229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:55.474233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:55.474366 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:55.475002 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:55.475007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:55.475011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:55.475284 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:55.475450 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:55.476319 (+ 869us) server_negotiation.cc:299] Negotiation successful
0619 01:09:55.476493 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 01:09:55.478112 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.458496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:33763)
0619 01:09:55.459309 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:55.459360 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:55.461488 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:55.462101 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:55.462109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:55.462623 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:55.463180 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.463197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.464243 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.464249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.465757 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.465770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.466593 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.466601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.466831 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.467486 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:55.467515 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:55.470161 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:55.473551 (+ 3390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:55.473559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:55.473574 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:55.474100 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:55.474669 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:55.474672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:55.474674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:55.474863 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:55.475451 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:55.475461 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:55.475896 (+ 435us) client_negotiation.cc:769] Sending connection context
0619 01:09:55.476212 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 01:09:55.476755 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":741,"threads_started":1}
I20260619 01:09:55.479651 15263 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41331
I20260619 01:09:55.480020 15263 heartbeater.cc:464] Registering TS with master...
I20260619 01:09:55.480756 15263 heartbeater.cc:511] Master 127.0.97.62:41331 requested a full tablet report, sending...
I20260619 01:09:55.482244 14815 ts_manager.cc:194] Registered new tserver with Master: 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661)
I20260619 01:09:55.483039 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33763
I20260619 01:09:55.487952 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:09:55.494632 15002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.488887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.0.97.62:41331)
0619 01:09:55.489124 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:55.489130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:55.489451 (+ 321us) server_negotiation.cc:407] Connection header received
0619 01:09:55.489711 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:55.489716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:55.489918 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:55.490046 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:09:55.490422 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.491172 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.492569 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.492973 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.493068 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.493582 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:09:55.494096 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:09:55.494202 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:55.494273 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 01:09:55.494335 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 01:09:55.495636 14815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:09:55.503083 14815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:09:55.538969 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.517385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.0.97.2:38193)
0619 01:09:55.523968 (+ 6583us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:55.523980 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:55.530081 (+ 6101us) server_negotiation.cc:407] Connection header received
0619 01:09:55.530362 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:55.530372 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:55.530619 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:55.530789 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:55.531922 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.533384 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.536325 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.537835 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.537984 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.538187 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:55.538450 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 01:09:55.538637 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6322,"thread_start_us":6200,"threads_started":1}
I20260619 01:09:55.539189 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.516908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.0.97.1:38697)
0619 01:09:55.529866 (+ 12958us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:55.529875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:55.529906 (+ 31us) server_negotiation.cc:407] Connection header received
0619 01:09:55.530017 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:55.530028 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:55.530321 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:55.530551 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:55.531730 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.533348 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.535822 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.537205 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.537386 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.537633 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:55.538677 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0619 01:09:55.538847 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12765,"thread_start_us":12604,"threads_started":1}
I20260619 01:09:55.539008 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.516262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38193 (local address 127.0.0.1:34232)
0619 01:09:55.529949 (+ 13687us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:55.529966 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:55.530180 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:55.531105 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:55.531109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:55.531128 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:55.531732 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.531746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.533553 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.533557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.536150 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.536164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.538077 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.538083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.538247 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.538284 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 01:09:55.538388 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 01:09:55.538561 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13569,"mutex_wait_us":146,"thread_start_us":11764,"threads_started":1}
I20260619 01:09:55.539773 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.515679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42661 (local address 127.0.0.1:50384)
0619 01:09:55.527394 (+ 11715us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:55.527468 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:55.527740 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:55.532205 (+ 4465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:55.532210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:55.532238 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:55.532556 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.532566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.534545 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.534550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.536718 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.536732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.538541 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.538546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.538722 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.538756 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 01:09:55.538819 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 01:09:55.538958 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11332,"thread_start_us":13335,"threads_started":1}
I20260619 01:09:55.539923 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.515883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.0.97.3:42661)
0619 01:09:55.531632 (+ 15749us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:55.531642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:55.531676 (+ 34us) server_negotiation.cc:407] Connection header received
0619 01:09:55.531771 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:55.531775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:55.532001 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:55.532202 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:55.532795 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.534388 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.536913 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.538307 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.539117 (+ 810us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.539342 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:55.539417 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 01:09:55.539579 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15565,"thread_start_us":15410,"threads_started":1}
I20260619 01:09:55.540035 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.516018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.0.1:48684)
0619 01:09:55.529317 (+ 13299us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:55.529340 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:55.529478 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:55.530881 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:55.530888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:55.530919 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:55.531497 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.531511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.533522 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.533527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.535650 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.535664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.537787 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.537793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.538034 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.538093 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 01:09:55.538225 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 01:09:55.538441 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13144,"mutex_wait_us":97,"thread_start_us":13799,"threads_started":1}
I20260619 01:09:55.543588 15198 tablet_service.cc:1467] Processing CreateTablet for tablet 314ecb73f72d457e97c489b79bc115d1 (DEFAULT_TABLE table=test-table [id=6171f8eab04f40f091f418b93dc706da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:09:55.543730 15067 tablet_service.cc:1467] Processing CreateTablet for tablet 314ecb73f72d457e97c489b79bc115d1 (DEFAULT_TABLE table=test-table [id=6171f8eab04f40f091f418b93dc706da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:09:55.543730 14935 tablet_service.cc:1467] Processing CreateTablet for tablet 314ecb73f72d457e97c489b79bc115d1 (DEFAULT_TABLE table=test-table [id=6171f8eab04f40f091f418b93dc706da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:09:55.544925 15198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314ecb73f72d457e97c489b79bc115d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:55.545046 15067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314ecb73f72d457e97c489b79bc115d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:55.545074 14935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 314ecb73f72d457e97c489b79bc115d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:09:55.551249 15000 heartbeater.cc:503] Master 127.0.97.62:41331 was elected leader, sending a full tablet report...
I20260619 01:09:55.558316 15279 tablet_bootstrap.cc:492] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: Bootstrap starting.
I20260619 01:09:55.561510 15280 tablet_bootstrap.cc:492] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Bootstrap starting.
I20260619 01:09:55.562363 15281 tablet_bootstrap.cc:492] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: Bootstrap starting.
I20260619 01:09:55.563532 15279 tablet_bootstrap.cc:654] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:55.564884 15279 log.cc:824] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:55.565069 15281 tablet_bootstrap.cc:654] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:55.565100 15280 tablet_bootstrap.cc:654] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Neither blocks nor log segments found. Creating new log.
I20260619 01:09:55.566202 15280 log.cc:824] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:55.566368 15281 log.cc:824] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:55.568218 15279 tablet_bootstrap.cc:492] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: No bootstrap required, opened a new log
I20260619 01:09:55.568536 15279 ts_tablet_manager.cc:1397] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260619 01:09:55.569550 15280 tablet_bootstrap.cc:492] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: No bootstrap required, opened a new log
I20260619 01:09:55.569830 15280 ts_tablet_manager.cc:1397] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 01:09:55.575434 15281 tablet_bootstrap.cc:492] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: No bootstrap required, opened a new log
I20260619 01:09:55.575780 15281 ts_tablet_manager.cc:1397] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260619 01:09:55.580510 15279 raft_consensus.cc:348] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:55.580577 15280 raft_consensus.cc:348] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:55.581035 15280 raft_consensus.cc:374] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:55.581038 15279 raft_consensus.cc:374] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:55.581135 15280 raft_consensus.cc:729] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a5caac1f564823b07ee4f7c9423544, State: Initialized, Role: FOLLOWER
I20260619 01:09:55.581135 15279 raft_consensus.cc:729] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f0274ee5733401bbd3348400b0dafdc, State: Initialized, Role: FOLLOWER
I20260619 01:09:55.581862 15280 consensus_queue.cc:260] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:55.581916 15279 consensus_queue.cc:260] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:55.584370 15281 raft_consensus.cc:348] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:55.584776 15281 raft_consensus.cc:374] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:09:55.584893 15281 raft_consensus.cc:729] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce5e73e194143a7bbada5b3f08dcd15, State: Initialized, Role: FOLLOWER
I20260619 01:09:55.585568 15281 consensus_queue.cc:260] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:55.586923 15132 heartbeater.cc:503] Master 127.0.97.62:41331 was elected leader, sending a full tablet report...
I20260619 01:09:55.587374 15281 ts_tablet_manager.cc:1428] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 01:09:55.589140 15279 ts_tablet_manager.cc:1428] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: Time spent starting tablet: real 0.021s user 0.015s sys 0.005s
I20260619 01:09:55.589447 15280 ts_tablet_manager.cc:1428] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260619 01:09:55.589643 15263 heartbeater.cc:503] Master 127.0.97.62:41331 was elected leader, sending a full tablet report...
W20260619 01:09:55.706905 15264 tablet.cc:2369] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:09:55.734193 15133 tablet.cc:2369] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:09:55.761153 15001 tablet.cc:2369] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:09:55.787437 15287 raft_consensus.cc:482] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:09:55.787781 15287 raft_consensus.cc:504] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:55.789592 15287 leader_election.cc:288] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661), 92a5caac1f564823b07ee4f7c9423544 (127.0.97.1:38697)
I20260619 01:09:55.796540 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.789917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52943 (local address 127.0.97.3:42661)
0619 01:09:55.790127 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:55.790134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:55.790765 (+ 631us) server_negotiation.cc:407] Connection header received
0619 01:09:55.791001 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:55.791008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:55.791097 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:55.791164 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:55.791823 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.792587 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.794679 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.795475 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.795638 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.795755 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:55.796148 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 01:09:55.796282 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 01:09:55.796537 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.790091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44519 (local address 127.0.97.1:38697)
0619 01:09:55.790306 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:55.790313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:55.790922 (+ 609us) server_negotiation.cc:407] Connection header received
0619 01:09:55.791084 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:55.791088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:55.791174 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:55.791268 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:09:55.791930 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.792631 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.794647 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.795444 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.795598 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.795727 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:55.796109 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 01:09:55.796262 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 01:09:55.796691 15288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.789769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42661 (local address 127.0.97.2:52943)
0619 01:09:55.790605 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:55.790629 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:55.790817 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:55.791214 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:55.791221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:55.791253 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:55.791664 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.791676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.792722 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.792730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.794479 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.794492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.795746 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.795755 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.795943 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.795982 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 01:09:55.796094 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 01:09:55.796263 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":131,"thread_start_us":385,"threads_started":1}
I20260619 01:09:55.797106 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.789917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.97.2:44519)
0619 01:09:55.790793 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:55.790812 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:55.790958 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:55.791305 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:55.791313 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:55.791350 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:09:55.791750 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.791766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.792790 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.792793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.794469 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:55.794488 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.795709 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.795716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:55.795929 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.795965 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 01:09:55.796043 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 01:09:55.796227 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":104,"thread_start_us":393,"threads_started":1}
I20260619 01:09:55.797937 15218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314ecb73f72d457e97c489b79bc115d1" candidate_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f0274ee5733401bbd3348400b0dafdc" is_pre_election: true
I20260619 01:09:55.798146 14955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314ecb73f72d457e97c489b79bc115d1" candidate_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a5caac1f564823b07ee4f7c9423544" is_pre_election: true
I20260619 01:09:55.798614 15218 raft_consensus.cc:2454] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ce5e73e194143a7bbada5b3f08dcd15 in term 0.
I20260619 01:09:55.798710 14955 raft_consensus.cc:2454] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ce5e73e194143a7bbada5b3f08dcd15 in term 0.
I20260619 01:09:55.799435 15022 leader_election.cc:302] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ce5e73e194143a7bbada5b3f08dcd15, 7f0274ee5733401bbd3348400b0dafdc; no voters:
I20260619 01:09:55.799925 15287 raft_consensus.cc:2781] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:09:55.800065 15287 raft_consensus.cc:482] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:09:55.800187 15287 raft_consensus.cc:3037] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:55.802742 15287 raft_consensus.cc:504] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:55.803359 15287 leader_election.cc:288] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [CANDIDATE]: Term 1 election: Requested vote from peers 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661), 92a5caac1f564823b07ee4f7c9423544 (127.0.97.1:38697)
I20260619 01:09:55.803943 14955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314ecb73f72d457e97c489b79bc115d1" candidate_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92a5caac1f564823b07ee4f7c9423544"
I20260619 01:09:55.803956 15218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "314ecb73f72d457e97c489b79bc115d1" candidate_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f0274ee5733401bbd3348400b0dafdc"
I20260619 01:09:55.804162 14955 raft_consensus.cc:3037] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:55.804189 15218 raft_consensus.cc:3037] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:09:55.806502 14955 raft_consensus.cc:2454] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ce5e73e194143a7bbada5b3f08dcd15 in term 1.
I20260619 01:09:55.806564 15218 raft_consensus.cc:2454] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ce5e73e194143a7bbada5b3f08dcd15 in term 1.
I20260619 01:09:55.806999 15020 leader_election.cc:302] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ce5e73e194143a7bbada5b3f08dcd15, 92a5caac1f564823b07ee4f7c9423544; no voters:
I20260619 01:09:55.807320 15287 raft_consensus.cc:2781] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:09:55.807724 15287 raft_consensus.cc:686] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 1 LEADER]: Becoming Leader. State: Replica: 4ce5e73e194143a7bbada5b3f08dcd15, State: Running, Role: LEADER
I20260619 01:09:55.808200 15287 consensus_queue.cc:237] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:55.813449 14814 catalog_manager.cc:5495] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } health_report { overall_health: HEALTHY } } }
I20260619 01:09:55.838541 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.833340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.0.97.2:38193)
0619 01:09:55.833536 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:55.833540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:55.833557 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:09:55.833678 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:55.833682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:55.833764 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:55.833883 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:09:55.834225 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.834948 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.836468 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.836865 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.836955 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.837438 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:09:55.837982 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:09:55.838096 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:55.838199 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 01:09:55.838268 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:09:55.846899 14955 raft_consensus.cc:1261] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Refusing update from remote peer 4ce5e73e194143a7bbada5b3f08dcd15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:09:55.846903 15218 raft_consensus.cc:1261] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc [term 1 FOLLOWER]: Refusing update from remote peer 4ce5e73e194143a7bbada5b3f08dcd15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:09:55.847968 15287 consensus_queue.cc:1035] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:09:55.848333 15290 consensus_queue.cc:1035] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:09:55.874874 14815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:09:55.875186 14815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:09:55.879118 14815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Sending DeleteTablet for 3 replicas of tablet 314ecb73f72d457e97c489b79bc115d1
I20260619 01:09:55.880421 15198 tablet_service.cc:1514] Processing DeleteTablet for tablet 314ecb73f72d457e97c489b79bc115d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:09:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50384
I20260619 01:09:55.880457 14935 tablet_service.cc:1514] Processing DeleteTablet for tablet 314ecb73f72d457e97c489b79bc115d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:09:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48684
I20260619 01:09:55.880929 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 14852
I20260619 01:09:55.881542 15067 tablet_service.cc:1514] Processing DeleteTablet for tablet 314ecb73f72d457e97c489b79bc115d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:09:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34232
I20260619 01:09:55.883126 15305 tablet_replica.cc:330] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: stopping tablet replica
I20260619 01:09:55.883637 15305 raft_consensus.cc:2229] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:09:55.886301 15305 raft_consensus.cc:2258] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:09:55.890333 15305 ts_tablet_manager.cc:1905] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:09:55.902614 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:55.881950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48694 (local address 127.0.97.1:38697)
0619 01:09:55.882139 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:55.882145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:55.882164 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:09:55.882268 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:55.882273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:55.882351 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:55.882448 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:55.884375 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.886018 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.887916 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:55.888892 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:55.889031 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:55.892542 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:55.892569 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:55.892596 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:55.892668 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:55.897912 (+ 5244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:55.898951 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:55.898962 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:55.898965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:55.899152 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:55.899713 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:55.899718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:55.899720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:55.900228 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:55.900530 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:55.901923 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0619 01:09:55.902179 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:09:55.903879 14995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:55.904783 15305 ts_tablet_manager.cc:1918] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:09:55.905009 15305 log.cc:1196] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/314ecb73f72d457e97c489b79bc115d1
I20260619 01:09:55.905678 15305 ts_tablet_manager.cc:1939] T 314ecb73f72d457e97c489b79bc115d1 P 4ce5e73e194143a7bbada5b3f08dcd15: Deleting consensus metadata
I20260619 01:09:55.905915 15304 tablet_replica.cc:330] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: stopping tablet replica
I20260619 01:09:55.906459 15304 raft_consensus.cc:2229] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:09:55.907086 15304 raft_consensus.cc:2258] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:09:55.907684 14800 catalog_manager.cc:4841] TS 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193): tablet 314ecb73f72d457e97c489b79bc115d1 (table test-table [id=6171f8eab04f40f091f418b93dc706da]) successfully deleted
I20260619 01:09:55.909904 15304 ts_tablet_manager.cc:1905] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:09:55.920149 15304 ts_tablet_manager.cc:1918] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:09:55.920390 15304 log.cc:1196] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/314ecb73f72d457e97c489b79bc115d1
I20260619 01:09:55.921111 15304 ts_tablet_manager.cc:1939] T 314ecb73f72d457e97c489b79bc115d1 P 7f0274ee5733401bbd3348400b0dafdc: Deleting consensus metadata
I20260619 01:09:55.923766 14801 catalog_manager.cc:4841] TS 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661): tablet 314ecb73f72d457e97c489b79bc115d1 (table test-table [id=6171f8eab04f40f091f418b93dc706da]) successfully deleted
I20260619 01:09:56.022923 15303 tablet_replica.cc:330] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: stopping tablet replica
I20260619 01:09:56.023192 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 14852
W20260619 01:09:56.043733 14799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38697 (error 108)
I20260619 01:09:56.044448 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 15004
I20260619 01:09:56.049894 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:56.047453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.0.1:48710)
0619 01:09:56.049549 (+ 2096us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:56.049691 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1961,"thread_start_us":1864,"threads_started":1}
W20260619 01:09:56.050201 14799 catalog_manager.cc:4584] TS 92a5caac1f564823b07ee4f7c9423544 (127.0.97.1:38697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 314ecb73f72d457e97c489b79bc115d1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38697: connect: Connection refused (error 111)
I20260619 01:09:56.061451 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:56.046250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.0.97.2:38193)
0619 01:09:56.046434 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:56.046440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:56.046468 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:09:56.046537 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:56.046542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:56.046633 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:56.046739 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:56.048617 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:56.049906 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:56.051579 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:56.052024 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:56.052202 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:56.054968 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:56.054984 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:56.055002 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:56.055043 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:56.058363 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:56.059145 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:56.059160 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:56.059163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:56.059309 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:56.059727 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:56.059732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:56.059735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:56.060101 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:56.060463 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:56.060702 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 01:09:56.060921 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 01:09:56.062183 15127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:56.104054 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:56.103584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.0.1:48720)
0619 01:09:56.103813 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:56.103913 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 01:09:56.105386 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:56.104889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.0.1:48722)
0619 01:09:56.105087 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:56.105171 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 01:09:56.166097 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:56.165636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.0.1:48726)
0619 01:09:56.165860 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:56.165949 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 01:09:56.167219 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:56.166852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.0.1:48740)
0619 01:09:56.167017 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:56.167086 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 01:09:56.177912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 15004
I20260619 01:09:56.197994 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 15135
I20260619 01:09:56.212028 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:56.198784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.0.97.3:42661)
0619 01:09:56.198984 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:56.198991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:56.199085 (+ 94us) server_negotiation.cc:407] Connection header received
0619 01:09:56.199305 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:56.199315 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:56.199447 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:56.199553 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:56.200850 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:56.201672 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:56.203053 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:56.203458 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:56.203611 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:56.206267 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:56.206283 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:56.206298 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:56.206334 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:56.208889 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:56.209728 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:56.209736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:56.209738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:56.209900 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:56.210291 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:56.210297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:56.210300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:56.210694 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:56.211015 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:56.211411 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 01:09:56.211612 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:09:56.212739 15258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:56.247699 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:56.247200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.0.1:48754)
0619 01:09:56.247440 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:56.247550 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 01:09:56.248903 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:56.248447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.0.1:48760)
0619 01:09:56.248682 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:56.248757 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 01:09:56.303938 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 15135
I20260619 01:09:56.321425 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 14782
I20260619 01:09:56.322676 14845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:09:56.438874 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:56.438202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38697 (local address 127.0.0.1:48768)
0619 01:09:56.438451 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:56.438590 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 01:09:56.439002 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 14782
I20260619 01:09:56.461561 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41331
--webserver_interface=127.0.97.62
--webserver_port=34357
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:09:56.805903 15312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:56.806221 15312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:56.806387 15312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:56.814049 15312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:09:56.814168 15312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:56.814275 15312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:56.814371 15312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:09:56.814450 15312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:09:56.824958 15312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:56.826506 15312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:56.828009 15312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:56.840745 15317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:56.840655 15318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:56.840639 15320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:56.841933 15312 server_base.cc:1034] running on GCE node
I20260619 01:09:56.842842 15312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:56.844462 15312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:56.845746 15312 hybrid_clock.cc:648] HybridClock initialized: now 1781831396845670 us; error 66 us; skew 500 ppm
I20260619 01:09:56.846184 15312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:56 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:56.849676 15312 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:09:56.851544 15312 webserver.cc:460] Webserver started at http://127.0.97.62:34357/ using document root <none> and password file <none>
I20260619 01:09:56.852427 15312 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:56.852622 15312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:56.859643 15312 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:09:56.863271 15327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:56.864825 15312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:09:56.865031 15312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "3828e0ed77c54d37ae263f79749ec113"
format_stamp: "Formatted at 2026-06-19 01:09:50 on dist-test-slave-5p8w"
server_key: "936bfed2ab5c9fb58ded52da8474801e"
server_key_iv: "d597f8bd717aa4ca0e1f108196020688"
server_key_version: "encryptionkey@0"
I20260619 01:09:56.866186 15312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:56.879316 15312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:56.884145 15312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:56.884460 15312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:56.905037 15312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41331
I20260619 01:09:56.905095 15378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41331 every 8 connection(s)
I20260619 01:09:56.906823 15312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:09:56.914458 15379 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:09:56.915798 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 15312
I20260619 01:09:56.916630 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38697
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41331
--webserver_port=40731
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:09:56.918341 15379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Bootstrap starting.
I20260619 01:09:56.947870 15379 log.cc:824] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:56.970544 15379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:09:56.971323 15379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113: Bootstrap complete.
I20260619 01:09:56.981000 15379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:56.982097 15379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3828e0ed77c54d37ae263f79749ec113, State: Initialized, Role: FOLLOWER
I20260619 01:09:56.982756 15379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:56.983024 15379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:09:56.983161 15379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:09:56.983350 15379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 2 FOLLOWER]: Advancing to term 3
I20260619 01:09:56.987526 15379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:56.988078 15379 leader_election.cc:302] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3828e0ed77c54d37ae263f79749ec113; no voters:
I20260619 01:09:56.988803 15379 leader_election.cc:288] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 01:09:56.988955 15383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 01:09:56.990208 15383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [term 3 LEADER]: Becoming Leader. State: Replica: 3828e0ed77c54d37ae263f79749ec113, State: Running, Role: LEADER
I20260619 01:09:56.990795 15383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } }
I20260619 01:09:56.991231 15379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:09:56.993441 15385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3828e0ed77c54d37ae263f79749ec113. Latest consensus state: current_term: 3 leader_uuid: "3828e0ed77c54d37ae263f79749ec113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } } }
I20260619 01:09:56.993846 15385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:56.994395 15384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3828e0ed77c54d37ae263f79749ec113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3828e0ed77c54d37ae263f79749ec113" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41331 } } }
I20260619 01:09:56.994736 15384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3828e0ed77c54d37ae263f79749ec113 [sys.catalog]: This master's current role is: LEADER
I20260619 01:09:57.004494 15390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:09:57.016391 15390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:09:57.018985 15390 catalog_manager.cc:1259] Loaded cluster ID: 05f35f0f09814adc82b44fdd58972148
I20260619 01:09:57.019151 15390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 01:09:57.022215 15390 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8ac06239ad google::LogMessage::Fail() at ??:0
@ 0x7f8ac062688a google::LogMessage::SendToLog() at ??:0
@ 0x7f8ac06234de google::LogMessage::Flush() at ??:0
@ 0x7f8ac0624a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8ad6056512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f8ad6053976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8ad60db610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f8ad60db439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8ac326f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8ac327291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8ac3272799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8ac324deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8ac65276db start_thread at ??:0
@ 0x7f8abf4bd61f clone at ??:0
W20260619 01:09:57.353359 15381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:57.353700 15381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:57.353864 15381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:57.365458 15381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:57.365613 15381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:57.365790 15381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:09:57.376616 15381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40731
--tserver_master_addrs=127.0.97.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:57.378108 15381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:57.379519 15381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:57.393980 15423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:57.394068 15421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:57.394019 15420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:57.400676 15381 server_base.cc:1034] running on GCE node
I20260619 01:09:57.401584 15381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:57.402740 15381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:57.404004 15381 hybrid_clock.cc:648] HybridClock initialized: now 1781831397403948 us; error 52 us; skew 500 ppm
I20260619 01:09:57.404374 15381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:57 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:57.408062 15381 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:09:57.409977 15381 webserver.cc:460] Webserver started at http://127.0.97.1:40731/ using document root <none> and password file <none>
I20260619 01:09:57.410809 15381 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:57.410923 15381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:57.417157 15381 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 01:09:57.420518 15430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:57.422140 15381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:09:57.422343 15381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "92a5caac1f564823b07ee4f7c9423544"
format_stamp: "Formatted at 2026-06-19 01:09:51 on dist-test-slave-5p8w"
server_key: "79e820ce1e371ae4161143c6718bce63"
server_key_iv: "bea7adee200fe63b36e54314692c5a32"
server_key_version: "encryptionkey@0"
I20260619 01:09:57.423414 15381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:57.437165 15381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:57.441658 15381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:57.441955 15381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:57.443065 15381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:57.444991 15437 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 01:09:57.451470 15381 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 01:09:57.451632 15381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260619 01:09:57.451814 15381 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 01:09:57.455331 15381 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 01:09:57.455432 15381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260619 01:09:57.455878 15437 tablet_bootstrap.cc:492] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Bootstrap starting.
I20260619 01:09:57.486232 15437 log.cc:824] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Log is configured to *not* fsync() on all Append() calls
I20260619 01:09:57.499838 15381 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38697
I20260619 01:09:57.499886 15544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38697 every 8 connection(s)
I20260619 01:09:57.501645 15381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:09:57.501719 15437 tablet_bootstrap.cc:492] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:09:57.502624 15437 tablet_bootstrap.cc:492] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Bootstrap complete.
I20260619 01:09:57.504326 15437 ts_tablet_manager.cc:1397] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Time spent bootstrapping tablet: real 0.049s user 0.035s sys 0.011s
I20260619 01:09:57.507735 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:57.504947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:46191)
0619 01:09:57.505951 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:57.506232 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":424,"threads_started":1}
I20260619 01:09:57.508172 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 15381
W20260619 01:09:57.508616 15545 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41331 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41331: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
I20260619 01:09:57.509006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38193
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41331
--webserver_port=46095
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:09:57.509929 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:57.509356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:51321)
0619 01:09:57.509643 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:57.509742 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 01:09:57.511130 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:57.510648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:54519)
0619 01:09:57.510831 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:57.510936 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 01:09:57.511509 15545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 01:09:57.513787 15437 raft_consensus.cc:348] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:57.514874 15437 raft_consensus.cc:729] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a5caac1f564823b07ee4f7c9423544, State: Initialized, Role: FOLLOWER
I20260619 01:09:57.515550 15437 consensus_queue.cc:260] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:57.517442 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:57.516984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:60789)
0619 01:09:57.517159 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:57.517247 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 01:09:57.517618 15437 ts_tablet_manager.cc:1428] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 01:09:57.518597 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:57.518239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:59375)
0619 01:09:57.518397 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:57.518476 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 01:09:57.864925 15548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:57.865309 15548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:57.865515 15548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:57.873417 15548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:57.873582 15548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:57.873773 15548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:09:57.884115 15548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46095
--tserver_master_addrs=127.0.97.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:57.885676 15548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:57.887152 15548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:57.901950 15554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:57.901903 15555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:57.901885 15557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:57.902895 15548 server_base.cc:1034] running on GCE node
I20260619 01:09:57.903707 15548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:57.904768 15548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:57.906064 15548 hybrid_clock.cc:648] HybridClock initialized: now 1781831397906004 us; error 51 us; skew 500 ppm
I20260619 01:09:57.906457 15548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:57 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:57.915642 15548 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:09:57.917438 15548 webserver.cc:460] Webserver started at http://127.0.97.2:46095/ using document root <none> and password file <none>
I20260619 01:09:57.918257 15548 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:57.918435 15548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:57.924839 15548 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 01:09:57.928469 15564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:57.930212 15548 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260619 01:09:57.930425 15548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "4ce5e73e194143a7bbada5b3f08dcd15"
format_stamp: "Formatted at 2026-06-19 01:09:51 on dist-test-slave-5p8w"
server_key: "418ff0fc1bfa629f520aa3a7bcb69de6"
server_key_iv: "84546008f258ddb79cb8eeb64e99ee36"
server_key_version: "encryptionkey@0"
I20260619 01:09:57.931563 15548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:57.947289 15548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:57.952399 15548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:57.952720 15548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:57.953899 15548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:57.955503 15548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:57.955658 15548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:57.955796 15548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:57.955885 15548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:57.991217 15548 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38193
I20260619 01:09:57.991305 15676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38193 every 8 connection(s)
I20260619 01:09:57.992800 15548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:09:57.995959 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 15548
I20260619 01:09:57.996592 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42661
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41331
--webserver_port=42191
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:09:57.998493 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:57.996212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:39577)
0619 01:09:57.997148 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:57.997442 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":398,"threads_started":1}
W20260619 01:09:57.999292 15677 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41331 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41331: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
I20260619 01:09:58.000324 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:57.999969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:40157)
0619 01:09:58.000134 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:58.000210 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 01:09:58.001232 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:58.000881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:50557)
0619 01:09:58.001031 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:58.001118 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 01:09:58.001504 15677 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:09:58.345641 15680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:09:58.346009 15680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:09:58.346189 15680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:09:58.353778 15680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:09:58.353958 15680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:09:58.354161 15680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:09:58.364243 15680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42191
--tserver_master_addrs=127.0.97.62:41331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:09:58.365692 15680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:09:58.367089 15680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:09:58.381141 15688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:58.381163 15685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:09:58.381176 15686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:09:58.382076 15680 server_base.cc:1034] running on GCE node
I20260619 01:09:58.382758 15680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:09:58.383749 15680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:09:58.384954 15680 hybrid_clock.cc:648] HybridClock initialized: now 1781831398384909 us; error 44 us; skew 500 ppm
I20260619 01:09:58.385397 15680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:09:58 dist-test-slave-5p8w krb5kdc[14230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:09:58.388705 15680 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:09:58.390578 15680 webserver.cc:460] Webserver started at http://127.0.97.3:42191/ using document root <none> and password file <none>
I20260619 01:09:58.391310 15680 fs_manager.cc:359] Metadata directory not provided
I20260619 01:09:58.391418 15680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:09:58.397114 15680 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 01:09:58.400142 15695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:58.401723 15680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:09:58.401922 15680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "7f0274ee5733401bbd3348400b0dafdc"
format_stamp: "Formatted at 2026-06-19 01:09:52 on dist-test-slave-5p8w"
server_key: "7fdf77b01533e9c2d6ac211d66c15fbd"
server_key_iv: "e2df793f1f348476a6d9b6885a80acec"
server_key_version: "encryptionkey@0"
I20260619 01:09:58.402983 15680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:09:58.429678 15680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:09:58.434670 15680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:09:58.434968 15680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:09:58.436146 15680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:09:58.437767 15680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:09:58.437904 15680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:58.438045 15680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:09:58.438133 15680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:09:58.472419 15680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42661
I20260619 01:09:58.472510 15807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42661 every 8 connection(s)
I20260619 01:09:58.474017 15680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:09:58.479128 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:58.477154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:54435)
0619 01:09:58.477964 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:58.478197 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":300,"threads_started":1}
W20260619 01:09:58.479919 15808 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41331 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41331: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
I20260619 01:09:58.480906 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:58.480524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:43425)
0619 01:09:58.480707 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:58.480788 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 01:09:58.481942 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:58.481629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:41061)
0619 01:09:58.481760 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:58.481834 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 01:09:58.482252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 15680
W20260619 01:09:58.482265 15808 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 01:09:58.520345 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:58.519553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:50577)
0619 01:09:58.520027 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:58.520133 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":226,"threads_started":1}
I20260619 01:09:58.892629 15813 raft_consensus.cc:482] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:09:58.892963 15813 raft_consensus.cc:504] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:09:58.894651 15813 leader_election.cc:288] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661), 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193)
Jun 19 01:09:58 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Jun 19 01:09:58 dist-test-slave-5p8w krb5kdc[14230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:09:58.918931 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:58.895385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54637 (local address 127.0.97.3:42661)
0619 01:09:58.897641 (+ 2256us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:58.897664 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:58.897698 (+ 34us) server_negotiation.cc:407] Connection header received
0619 01:09:58.899162 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:58.899170 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:58.899590 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:58.899964 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:58.902218 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:58.903456 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:58.905597 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:58.906259 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:58.906465 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:58.911274 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:58.911282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:58.911295 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:58.911338 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:58.914274 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:58.915293 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:58.915304 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:58.915314 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:58.915542 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:58.916131 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:58.916137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:58.916142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:58.916885 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:58.917113 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:58.918043 (+ 930us) server_negotiation.cc:299] Negotiation successful
0619 01:09:58.918494 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2049,"thread_start_us":1865,"threads_started":1}
I20260619 01:09:58.918972 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:58.895030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38193 (local address 127.0.97.1:44437)
0619 01:09:58.897356 (+ 2326us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:58.897383 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:58.897524 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:58.900139 (+ 2615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:58.900150 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:58.901054 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:58.901952 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:58.901975 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:58.903816 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:58.903821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:58.905443 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:58.905456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:58.906526 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:58.906532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:58.906876 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:58.907575 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:58.907598 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:58.910879 (+ 3281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:58.914949 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:58.914953 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:58.914955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:58.915901 (+ 946us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:58.916453 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:58.916456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:58.916458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:58.916581 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:58.917603 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:58.917608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:58.917758 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 01:09:58.917906 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 01:09:58.918495 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2158,"mutex_wait_us":196,"spinlock_wait_cycles":1920,"thread_start_us":1991,"threads_started":1}
I20260619 01:09:58.919023 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:58.894951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42661 (local address 127.0.97.1:54637)
0619 01:09:58.895234 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:58.895289 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 01:09:58.895940 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:09:58.900139 (+ 4199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:09:58.900150 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:09:58.901062 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:09:58.901952 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:58.901974 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:58.903712 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:58.903721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:58.905440 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:09:58.905455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:09:58.906510 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:09:58.906517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:09:58.906848 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:58.907760 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:09:58.907779 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:09:58.910976 (+ 3197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:09:58.914454 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:58.914464 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:58.914482 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:58.915106 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:58.915713 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:09:58.915723 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:09:58.915727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:09:58.916006 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:09:58.917025 (+ 1019us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:09:58.917032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:09:58.917426 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 01:09:58.917806 (+ 380us) client_negotiation.cc:240] Negotiation successful
0619 01:09:58.918626 (+ 820us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":405,"spinlock_wait_cycles":26752}
I20260619 01:09:58.919230 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:58.895328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44437 (local address 127.0.97.2:38193)
0619 01:09:58.896847 (+ 1519us) server_negotiation.cc:206] Beginning negotiation
0619 01:09:58.896866 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 01:09:58.897522 (+ 656us) server_negotiation.cc:407] Connection header received
0619 01:09:58.899086 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:09:58.899099 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:09:58.899641 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:09:58.899992 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:09:58.902239 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:58.903627 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:58.905609 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:09:58.906262 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:09:58.906476 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:09:58.911246 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:09:58.911255 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:09:58.911272 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:09:58.911315 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:09:58.914743 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:58.916074 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:58.916084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:58.916094 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:58.916333 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:09:58.916704 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:09:58.916710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:09:58.916713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:09:58.917398 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:09:58.917614 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 01:09:58.918320 (+ 706us) server_negotiation.cc:299] Negotiation successful
0619 01:09:58.918745 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1317,"thread_start_us":1185,"threads_started":1}
W20260619 01:09:58.923115 15432 leader_election.cc:341] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193): Not found: Tablet not found: 314ecb73f72d457e97c489b79bc115d1
W20260619 01:09:58.923573 15433 leader_election.cc:341] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661): Not found: Tablet not found: 314ecb73f72d457e97c489b79bc115d1
I20260619 01:09:58.923748 15433 leader_election.cc:302] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92a5caac1f564823b07ee4f7c9423544; no voters: 4ce5e73e194143a7bbada5b3f08dcd15, 7f0274ee5733401bbd3348400b0dafdc
I20260619 01:09:58.924188 15813 raft_consensus.cc:2726] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 01:09:59.003268 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:59.002234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:59021)
0619 01:09:59.002811 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:59.002982 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":273,"threads_started":1}
I20260619 01:09:59.483785 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:59.482874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:47949)
0619 01:09:59.483408 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:59.483564 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":310,"threads_started":1}
I20260619 01:09:59.522302 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:09:59.521573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:41179)
0619 01:09:59.522036 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 01:09:59.522132 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":235,"threads_started":1}
I20260619 01:10:00.005414 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:00.004602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:39095)
0619 01:10:00.005127 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:00.005222 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":270,"threads_started":1}
I20260619 01:10:00.485798 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:00.484864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:37335)
0619 01:10:00.485457 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:00.485573 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":275,"threads_started":1}
I20260619 01:10:00.524106 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:00.523288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:44967)
0619 01:10:00.523816 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:00.523900 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":224,"threads_started":1}
I20260619 01:10:00.985497 15823 raft_consensus.cc:482] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:10:00.985715 15823 raft_consensus.cc:504] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:10:00.986421 15823 leader_election.cc:288] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661), 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193)
W20260619 01:10:00.987648 15432 leader_election.cc:341] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193): Not found: Tablet not found: 314ecb73f72d457e97c489b79bc115d1
W20260619 01:10:00.987901 15433 leader_election.cc:341] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661): Not found: Tablet not found: 314ecb73f72d457e97c489b79bc115d1
I20260619 01:10:00.988049 15433 leader_election.cc:302] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92a5caac1f564823b07ee4f7c9423544; no voters: 4ce5e73e194143a7bbada5b3f08dcd15, 7f0274ee5733401bbd3348400b0dafdc
I20260619 01:10:00.988359 15823 raft_consensus.cc:2726] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 01:10:01.007382 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:01.006529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:57769)
0619 01:10:01.007121 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:01.007212 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":276,"threads_started":1}
I20260619 01:10:01.487776 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:01.486957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:52781)
0619 01:10:01.487503 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:01.487595 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":293,"threads_started":1}
I20260619 01:10:01.526194 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:01.525242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:57347)
0619 01:10:01.525912 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:01.526015 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":433,"threads_started":1}
I20260619 01:10:02.009537 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:02.008561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:42401)
0619 01:10:02.009138 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:02.009234 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":286,"threads_started":1}
I20260619 01:10:02.489728 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:02.488818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:52541)
0619 01:10:02.489436 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:02.489536 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":277,"threads_started":1}
I20260619 01:10:02.528546 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:02.527598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:36679)
0619 01:10:02.528190 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:02.528280 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":331,"threads_started":1}
I20260619 01:10:03.011627 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:03.010746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:49243)
0619 01:10:03.011302 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:03.011437 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":329,"threads_started":1}
I20260619 01:10:03.491906 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:03.491032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:35029)
0619 01:10:03.491580 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:03.491688 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":262,"threads_started":1}
I20260619 01:10:03.516400 15832 raft_consensus.cc:482] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:10:03.516636 15832 raft_consensus.cc:504] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:10:03.517402 15832 leader_election.cc:288] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661), 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193)
W20260619 01:10:03.518587 15433 leader_election.cc:341] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661): Not found: Tablet not found: 314ecb73f72d457e97c489b79bc115d1
W20260619 01:10:03.518837 15432 leader_election.cc:341] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193): Not found: Tablet not found: 314ecb73f72d457e97c489b79bc115d1
I20260619 01:10:03.518939 15432 leader_election.cc:302] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92a5caac1f564823b07ee4f7c9423544; no voters: 4ce5e73e194143a7bbada5b3f08dcd15, 7f0274ee5733401bbd3348400b0dafdc
I20260619 01:10:03.519230 15832 raft_consensus.cc:2726] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 01:10:03.530521 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:03.529566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:33761)
0619 01:10:03.530196 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:03.530311 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":380,"threads_started":1}
I20260619 01:10:04.013571 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:04.012706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:56137)
0619 01:10:04.013252 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:04.013382 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":292,"threads_started":1}
I20260619 01:10:04.493938 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:04.493043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:58079)
0619 01:10:04.493633 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:04.493747 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":353,"threads_started":1}
I20260619 01:10:04.532449 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:04.531616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:41327)
0619 01:10:04.532128 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:04.532243 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":269,"threads_started":1}
I20260619 01:10:05.015625 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:05.014662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:57509)
0619 01:10:05.015288 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:05.015398 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":290,"threads_started":1}
I20260619 01:10:05.495983 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:05.495033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:49535)
0619 01:10:05.495649 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:05.495755 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":281,"threads_started":1}
I20260619 01:10:05.534320 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:05.533553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:57565)
0619 01:10:05.534027 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:05.534121 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":236,"threads_started":1}
I20260619 01:10:05.731165 15840 raft_consensus.cc:482] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:10:05.731427 15840 raft_consensus.cc:504] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:10:05.732193 15840 leader_election.cc:288] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661), 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193)
W20260619 01:10:05.733364 15432 leader_election.cc:341] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193): Not found: Tablet not found: 314ecb73f72d457e97c489b79bc115d1
W20260619 01:10:05.733548 15433 leader_election.cc:341] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661): Not found: Tablet not found: 314ecb73f72d457e97c489b79bc115d1
I20260619 01:10:05.733731 15433 leader_election.cc:302] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92a5caac1f564823b07ee4f7c9423544; no voters: 4ce5e73e194143a7bbada5b3f08dcd15, 7f0274ee5733401bbd3348400b0dafdc
I20260619 01:10:05.734031 15840 raft_consensus.cc:2726] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 01:10:06.017679 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:06.016781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:33409)
0619 01:10:06.017387 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:06.017480 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":292,"threads_started":1}
I20260619 01:10:06.497961 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:06.497144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:39721)
0619 01:10:06.497660 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:06.497775 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":293,"threads_started":1}
I20260619 01:10:06.536088 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:06.535300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:37097)
0619 01:10:06.535824 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:06.535905 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":244,"threads_started":1}
I20260619 01:10:07.019505 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:07.018746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:36691)
0619 01:10:07.019257 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:07.019345 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":246,"threads_started":1}
I20260619 01:10:07.500057 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:07.498984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:59339)
0619 01:10:07.499667 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:07.499806 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":350,"threads_started":1}
I20260619 01:10:07.538108 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:07.537380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:42829)
0619 01:10:07.537839 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:07.537935 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
I20260619 01:10:08.021255 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:08.020498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:55969)
0619 01:10:08.020973 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:08.021062 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":244,"threads_started":1}
I20260619 01:10:08.502234 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:08.501350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:52315)
0619 01:10:08.501868 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:08.501982 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":240,"threads_started":1}
I20260619 01:10:08.539999 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:08.539155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:34409)
0619 01:10:08.539626 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:08.539774 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":233,"threads_started":1}
I20260619 01:10:09.023355 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:09.022366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:40363)
0619 01:10:09.023046 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:09.023171 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":383,"threads_started":1}
I20260619 01:10:09.504129 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:09.503272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:58159)
0619 01:10:09.503883 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:09.503972 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":322,"threads_started":1}
I20260619 01:10:09.541929 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:09.541076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:41683)
0619 01:10:09.541620 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:09.541712 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":307,"threads_started":1}
I20260619 01:10:10.025477 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:10.024554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:41517)
0619 01:10:10.025102 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:10.025203 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":298,"threads_started":1}
I20260619 01:10:10.506258 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:10.505214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:33889)
0619 01:10:10.505918 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:10.506011 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":393,"threads_started":1}
I20260619 01:10:10.543713 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:10.542939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:53565)
0619 01:10:10.543434 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:10.543528 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":248,"threads_started":1}
I20260619 01:10:11.027311 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:11.026506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:53099)
0619 01:10:11.026989 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:11.027082 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":238,"threads_started":1}
I20260619 01:10:11.508121 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:11.507245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:45049)
0619 01:10:11.507841 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:11.507939 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":341,"threads_started":1}
I20260619 01:10:11.545637 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:11.544802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:36209)
0619 01:10:11.545274 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:11.545437 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":236,"threads_started":1}
I20260619 01:10:12.029202 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:12.028290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:59297)
0619 01:10:12.028897 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:12.028987 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":346,"threads_started":1}
I20260619 01:10:12.260751 15860 raft_consensus.cc:482] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:10:12.261003 15860 raft_consensus.cc:504] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f0274ee5733401bbd3348400b0dafdc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42661 } } peers { permanent_uuid: "92a5caac1f564823b07ee4f7c9423544" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38697 } } peers { permanent_uuid: "4ce5e73e194143a7bbada5b3f08dcd15" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38193 } }
I20260619 01:10:12.261878 15860 leader_election.cc:288] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661), 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193)
W20260619 01:10:12.262760 15433 leader_election.cc:341] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f0274ee5733401bbd3348400b0dafdc (127.0.97.3:42661): Not found: Tablet not found: 314ecb73f72d457e97c489b79bc115d1
W20260619 01:10:12.262946 15432 leader_election.cc:341] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4ce5e73e194143a7bbada5b3f08dcd15 (127.0.97.2:38193): Not found: Tablet not found: 314ecb73f72d457e97c489b79bc115d1
I20260619 01:10:12.263123 15432 leader_election.cc:302] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92a5caac1f564823b07ee4f7c9423544; no voters: 4ce5e73e194143a7bbada5b3f08dcd15, 7f0274ee5733401bbd3348400b0dafdc
I20260619 01:10:12.263373 15860 raft_consensus.cc:2726] T 314ecb73f72d457e97c489b79bc115d1 P 92a5caac1f564823b07ee4f7c9423544 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 01:10:12.509903 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:12.509137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:40019)
0619 01:10:12.509609 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:12.509699 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":266,"threads_started":1}
I20260619 01:10:12.547780 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:12.546858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:39149)
0619 01:10:12.547467 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:12.547557 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":314,"threads_started":1}
I20260619 01:10:13.031301 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:13.030442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.2:55379)
0619 01:10:13.031035 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:13.031139 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":323,"threads_started":1}
I20260619 01:10:13.487232 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 15381
I20260619 01:10:13.504546 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:13.488686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.0.97.1:38697)
0619 01:10:13.489444 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:13.489455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:13.489491 (+ 36us) server_negotiation.cc:407] Connection header received
0619 01:10:13.489648 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:13.489656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:13.489993 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:13.490187 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:13.491875 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:13.493052 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:13.494547 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:13.495140 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:13.495249 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:13.498257 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:13.498272 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:13.498289 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:13.498334 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:13.501342 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:13.502277 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:13.502285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:13.502287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:13.502447 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:13.502882 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:13.502897 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:13.502899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:13.503282 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:13.503577 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:13.503934 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 01:10:13.504162 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":458,"threads_started":1}
I20260619 01:10:13.506058 15540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:10:13.511931 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:13.511113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.3:43879)
0619 01:10:13.511640 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:13.511728 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":265,"threads_started":1}
I20260619 01:10:13.600186 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:13.599115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41331 (local address 127.0.97.1:42371)
0619 01:10:13.599723 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:13.599867 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":309,"threads_started":1}
I20260619 01:10:13.600550 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 15381
I20260619 01:10:13.619158 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 15548
I20260619 01:10:13.634241 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:13.620008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.0.97.2:38193)
0619 01:10:13.620483 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:13.620488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:13.620505 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:10:13.620563 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:13.620566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:13.620629 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:13.620725 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:13.622354 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:13.623223 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:13.624592 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:13.625022 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:13.625127 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:13.628071 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:13.628077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:13.628081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:13.628126 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:13.631295 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:13.632165 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:13.632170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:13.632172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:13.632344 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:13.632775 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:13.632780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:13.632782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:13.633112 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:13.633449 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:13.633708 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 01:10:13.633934 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
I20260619 01:10:13.635099 15672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:10:13.723413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 15548
I20260619 01:10:13.742479 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 15680
I20260619 01:10:13.757056 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:13.743450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.0.97.3:42661)
0619 01:10:13.744049 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:13.744054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:13.744070 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:10:13.744125 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:13.744128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:13.744201 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:13.744338 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:13.745963 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:13.746820 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:13.748141 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:13.748589 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:13.748733 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:13.751551 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:13.751556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:13.751560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:13.751598 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:13.754306 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:13.754975 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:13.754982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:13.754985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:13.755123 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:13.755531 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:13.755539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:13.755542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:13.755815 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:13.756112 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:13.756452 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 01:10:13.756681 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":314,"threads_started":1}
I20260619 01:10:13.757884 15803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:10:13.844966 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 15680
2026-06-19T01:10:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24837 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:10:13 dist-test-slave-5p8w krb5kdc[15874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:10:13 dist-test-slave-5p8w krb5kdc[15874](info): set up 2 sockets
Jun 19 01:10:13 dist-test-slave-5p8w krb5kdc[15874](info): commencing operation
krb5kdc: starting...
W20260619 01:10:14.957932 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:10:15 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:10:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:10:15.221952 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38821
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:10:15.568325 15887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:10:15.568658 15887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:10:15.568837 15887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:10:15.576472 15887 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 01:10:15.576587 15887 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 01:10:15.576673 15887 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 01:10:15.576749 15887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:10:15.576848 15887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:10:15.576974 15887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:10:15.577040 15887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:10:15.577121 15887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:10:15.587514 15887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:10:15.588919 15887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:10:15.590379 15887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:10:15.603607 15892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:15.603626 15893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:15.603660 15895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:10:15.604533 15887 server_base.cc:1034] running on GCE node
I20260619 01:10:15.605400 15887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:10:15.606930 15887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:10:15.608192 15887 hybrid_clock.cc:648] HybridClock initialized: now 1781831415608131 us; error 32 us; skew 500 ppm
I20260619 01:10:15.608585 15887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:10:15 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:10:15.612018 15887 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:10:15.613716 15887 webserver.cc:460] Webserver started at http://127.0.97.62:36171/ using document root <none> and password file <none>
I20260619 01:10:15.614506 15887 fs_manager.cc:359] Metadata directory not provided
I20260619 01:10:15.614598 15887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:10:15.614950 15887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:10:15.618263 15887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "23df21a0905645d2a99a3b7e8c348ddb"
format_stamp: "Formatted at 2026-06-19 01:10:15 on dist-test-slave-5p8w"
server_key: "00ac694168a0d0e7afea4f52e4726776"
server_key_iv: "81b899a32bfb6b4bf52926a776567f4a"
server_key_version: "encryptionkey@0"
I20260619 01:10:15.619127 15887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "23df21a0905645d2a99a3b7e8c348ddb"
format_stamp: "Formatted at 2026-06-19 01:10:15 on dist-test-slave-5p8w"
server_key: "00ac694168a0d0e7afea4f52e4726776"
server_key_iv: "81b899a32bfb6b4bf52926a776567f4a"
server_key_version: "encryptionkey@0"
I20260619 01:10:15.624864 15887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 01:10:15.628732 15902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:15.630220 15887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:10:15.630420 15887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "23df21a0905645d2a99a3b7e8c348ddb"
format_stamp: "Formatted at 2026-06-19 01:10:15 on dist-test-slave-5p8w"
server_key: "00ac694168a0d0e7afea4f52e4726776"
server_key_iv: "81b899a32bfb6b4bf52926a776567f4a"
server_key_version: "encryptionkey@0"
I20260619 01:10:15.630679 15887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:10:15.660602 15887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:10:15.665009 15887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:10:15.665376 15887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:10:15.684800 15887 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38821
I20260619 01:10:15.684823 15953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38821 every 8 connection(s)
I20260619 01:10:15.686307 15887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:10:15.686800 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 15887
I20260619 01:10:15.687036 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:10:15.687920 388 external_mini_cluster.cc:1428] Setting key 2a86436b428afacd85c06578ce584d5c
I20260619 01:10:15.691005 15954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:10:15 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:10:15.702344 15954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Bootstrap starting.
I20260619 01:10:15.706230 15954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Neither blocks nor log segments found. Creating new log.
I20260619 01:10:15.707538 15954 log.cc:824] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Log is configured to *not* fsync() on all Append() calls
I20260619 01:10:15.711357 15954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: No bootstrap required, opened a new log
I20260619 01:10:15.712020 15957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:15.690074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.0.97.62:38821)
0619 01:10:15.690886 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:15.690905 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:15.690956 (+ 51us) server_negotiation.cc:407] Connection header received
0619 01:10:15.692717 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:15.692741 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:15.693179 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:15.693677 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:15.695187 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:15.696592 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:15.698145 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:15.698851 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:15.699092 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:15.703496 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:15.703505 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:15.703521 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:15.703563 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:15.707054 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:15.708072 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:15.708083 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:15.708100 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:15.708389 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:15.708853 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:15.708858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:15.708862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:15.709573 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:15.709816 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:15.710320 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 01:10:15.710704 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":368,"threads_started":1}
I20260619 01:10:15.720522 15954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:10:15.720984 15954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:10:15.721120 15954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23df21a0905645d2a99a3b7e8c348ddb, State: Initialized, Role: FOLLOWER
I20260619 01:10:15.721905 15954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:10:15.722131 15954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:10:15.722286 15954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:10:15.722486 15954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:10:15.725078 15954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:10:15.725765 15954 leader_election.cc:302] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23df21a0905645d2a99a3b7e8c348ddb; no voters:
I20260619 01:10:15.726397 15954 leader_election.cc:288] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:10:15.726538 15959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:10:15.727238 15959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 1 LEADER]: Becoming Leader. State: Replica: 23df21a0905645d2a99a3b7e8c348ddb, State: Running, Role: LEADER
I20260619 01:10:15.727815 15959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:10:15.728597 15954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:10:15.731079 15960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23df21a0905645d2a99a3b7e8c348ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } } }
I20260619 01:10:15.731056 15961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23df21a0905645d2a99a3b7e8c348ddb. Latest consensus state: current_term: 1 leader_uuid: "23df21a0905645d2a99a3b7e8c348ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } } }
I20260619 01:10:15.731597 15960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: This master's current role is: LEADER
I20260619 01:10:15.731608 15961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: This master's current role is: LEADER
I20260619 01:10:15.733505 15967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:10:15.738925 15967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:10:15.748994 15967 catalog_manager.cc:1347] Generated new cluster ID: c5b6de766ad44ac7b2518e413ec8af24
I20260619 01:10:15.749137 15967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:10:15.769899 15967 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:10:15.771690 15967 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:10:15.781528 15967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Generated new TSK 0
I20260619 01:10:15.782485 15967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:10:15.848835 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:10:16.203538 15980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:10:16.203845 15980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:10:16.203943 15980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:10:16.211637 15980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:10:16.211767 15980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:10:16.211887 15980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:10:16.222325 15980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:10:16.223649 15980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:10:16.224949 15980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:10:16.238935 15986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:16.239027 15985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:16.239681 15988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:10:16.239739 15980 server_base.cc:1034] running on GCE node
I20260619 01:10:16.240476 15980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:10:16.241557 15980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:10:16.242786 15980 hybrid_clock.cc:648] HybridClock initialized: now 1781831416242726 us; error 58 us; skew 500 ppm
I20260619 01:10:16.243137 15980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:10:16 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:10:16.251106 15980 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:10:16.252826 15980 webserver.cc:460] Webserver started at http://127.0.97.1:38783/ using document root <none> and password file <none>
I20260619 01:10:16.253647 15980 fs_manager.cc:359] Metadata directory not provided
I20260619 01:10:16.253763 15980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:10:16.254053 15980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:10:16.257071 15980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "1a992b7f6c9843ecbcfcc095cff9d893"
format_stamp: "Formatted at 2026-06-19 01:10:16 on dist-test-slave-5p8w"
server_key: "2926ba4e97fba9e014098dc4866f310e"
server_key_iv: "c4f81fd8745c188387eedcc6cd6073cc"
server_key_version: "encryptionkey@0"
I20260619 01:10:16.257920 15980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1a992b7f6c9843ecbcfcc095cff9d893"
format_stamp: "Formatted at 2026-06-19 01:10:16 on dist-test-slave-5p8w"
server_key: "2926ba4e97fba9e014098dc4866f310e"
server_key_iv: "c4f81fd8745c188387eedcc6cd6073cc"
server_key_version: "encryptionkey@0"
I20260619 01:10:16.263739 15980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:10:16.267817 15995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:16.269497 15980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:10:16.269658 15980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "1a992b7f6c9843ecbcfcc095cff9d893"
format_stamp: "Formatted at 2026-06-19 01:10:16 on dist-test-slave-5p8w"
server_key: "2926ba4e97fba9e014098dc4866f310e"
server_key_iv: "c4f81fd8745c188387eedcc6cd6073cc"
server_key_version: "encryptionkey@0"
I20260619 01:10:16.269845 15980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:10:16.289494 15980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:10:16.294292 15980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:10:16.294617 15980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:10:16.295661 15980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:10:16.297137 15980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:10:16.297212 15980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:16.297389 15980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:10:16.297454 15980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:16.332736 15980 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39611
I20260619 01:10:16.332787 16107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39611 every 8 connection(s)
I20260619 01:10:16.334401 15980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:10:16.344084 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 15980
I20260619 01:10:16.344329 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:10:16.345003 388 external_mini_cluster.cc:1428] Setting key 030c9064bdd183ca3e23a7eeac451b24
Jun 19 01:10:16 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:10:16.357146 16110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:16.337165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43043 (local address 127.0.97.62:38821)
0619 01:10:16.337637 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:16.337641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:16.338806 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 01:10:16.340784 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:16.340790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:16.340893 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:16.341028 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:16.342894 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:16.343667 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:16.345633 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:16.346207 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:16.346359 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:16.350570 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:16.350576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:16.350580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:16.350618 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:16.353594 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:16.354544 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:16.354550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:16.354552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:16.354682 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:16.355186 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:16.355191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:16.355193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:16.355483 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:16.355656 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:16.356487 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 01:10:16.356761 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20260619 01:10:16.358314 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:16.337837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:43043)
0619 01:10:16.338620 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:16.338666 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:16.340461 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:16.341368 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:16.341378 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:16.341997 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:10:16.342664 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:16.342687 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:16.343860 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:16.343864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:16.345465 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:16.345479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:16.346489 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:16.346500 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:16.346800 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:16.347647 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:10:16.347677 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:10:16.350274 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:10:16.353790 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:16.353801 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:16.353822 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:16.354368 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:16.354809 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:16.354814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:16.354818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:16.355045 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:16.355630 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:10:16.355639 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:10:16.356064 (+ 425us) client_negotiation.cc:769] Sending connection context
0619 01:10:16.356365 (+ 301us) client_negotiation.cc:240] Negotiation successful
0619 01:10:16.357007 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":297,"threads_started":1}
I20260619 01:10:16.360116 16108 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260619 01:10:16.360486 16108 heartbeater.cc:464] Registering TS with master...
I20260619 01:10:16.361385 16108 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260619 01:10:16.363894 15919 ts_manager.cc:194] Registered new tserver with Master: 1a992b7f6c9843ecbcfcc095cff9d893 (127.0.97.1:39611)
I20260619 01:10:16.365613 15919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43043
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:10:16.400544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:10:16.740677 16114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:10:16.741029 16114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:10:16.741205 16114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:10:16.749267 16114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:10:16.749461 16114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:10:16.749640 16114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:10:16.759781 16114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:10:16.761159 16114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:10:16.762621 16114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:10:16.778182 16120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:16.778607 16122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:16.778136 16119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:10:16.779469 16114 server_base.cc:1034] running on GCE node
I20260619 01:10:16.780174 16114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:10:16.781203 16114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:10:16.782475 16114 hybrid_clock.cc:648] HybridClock initialized: now 1781831416782399 us; error 75 us; skew 500 ppm
I20260619 01:10:16.782850 16114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:10:16 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:10:16.786268 16114 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:10:16.787863 16114 webserver.cc:460] Webserver started at http://127.0.97.2:34625/ using document root <none> and password file <none>
I20260619 01:10:16.788594 16114 fs_manager.cc:359] Metadata directory not provided
I20260619 01:10:16.788751 16114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:10:16.789083 16114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:10:16.792223 16114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "c79e1bb3913541228217340ebb478f76"
format_stamp: "Formatted at 2026-06-19 01:10:16 on dist-test-slave-5p8w"
server_key: "189b9452a1385f7cc7ea0e9367ecece7"
server_key_iv: "a8db51466d4918e3d5883e9d5a302f89"
server_key_version: "encryptionkey@0"
I20260619 01:10:16.793035 16114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c79e1bb3913541228217340ebb478f76"
format_stamp: "Formatted at 2026-06-19 01:10:16 on dist-test-slave-5p8w"
server_key: "189b9452a1385f7cc7ea0e9367ecece7"
server_key_iv: "a8db51466d4918e3d5883e9d5a302f89"
server_key_version: "encryptionkey@0"
I20260619 01:10:16.798569 16114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 01:10:16.802228 16129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:16.803623 16114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:10:16.803810 16114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "c79e1bb3913541228217340ebb478f76"
format_stamp: "Formatted at 2026-06-19 01:10:16 on dist-test-slave-5p8w"
server_key: "189b9452a1385f7cc7ea0e9367ecece7"
server_key_iv: "a8db51466d4918e3d5883e9d5a302f89"
server_key_version: "encryptionkey@0"
I20260619 01:10:16.804049 16114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:10:16.831067 16114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:10:16.836087 16114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:10:16.836390 16114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:10:16.837495 16114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:10:16.838985 16114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:10:16.839094 16114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:16.839244 16114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:10:16.839316 16114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:16.874089 16114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35715
I20260619 01:10:16.874107 16241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35715 every 8 connection(s)
I20260619 01:10:16.875514 16114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:10:16.885394 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 16114
I20260619 01:10:16.885607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:10:16.886053 388 external_mini_cluster.cc:1428] Setting key 32b1be788b127556edc024b94dc6c6cd
Jun 19 01:10:16 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:10:16.897505 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:16.878245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37217 (local address 127.0.97.62:38821)
0619 01:10:16.878705 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:16.878710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:16.879754 (+ 1044us) server_negotiation.cc:407] Connection header received
0619 01:10:16.881783 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:16.881790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:16.881890 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:16.882023 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:16.883695 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:16.884434 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:16.886328 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:16.886928 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:16.887087 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:16.891192 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:16.891198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:16.891201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:16.891240 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:16.894072 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:16.894932 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:16.894937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:16.894940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:16.895096 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:16.895580 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:16.895584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:16.895587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:16.895858 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:16.896037 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:16.896828 (+ 791us) server_negotiation.cc:299] Negotiation successful
0619 01:10:16.897080 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":232,"threads_started":1}
I20260619 01:10:16.898578 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:16.878810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:37217)
0619 01:10:16.879556 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:16.879602 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:16.881482 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:16.882284 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:16.882293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:16.882870 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:10:16.883493 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:16.883512 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:16.884615 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:16.884619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:16.886175 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:16.886190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:16.887203 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:16.887211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:16.887491 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:16.888248 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:10:16.888274 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:10:16.890885 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:10:16.894253 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:16.894268 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:16.894289 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:16.894778 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:16.895217 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:16.895222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:16.895226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:16.895430 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:16.896008 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:10:16.896017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:10:16.896400 (+ 383us) client_negotiation.cc:769] Sending connection context
0619 01:10:16.896718 (+ 318us) client_negotiation.cc:240] Negotiation successful
0619 01:10:16.897329 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":281,"threads_started":1}
I20260619 01:10:16.900170 16242 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260619 01:10:16.900563 16242 heartbeater.cc:464] Registering TS with master...
I20260619 01:10:16.901546 16242 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260619 01:10:16.903435 15919 ts_manager.cc:194] Registered new tserver with Master: c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715)
I20260619 01:10:16.904793 15919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37217
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:10:16.942436 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:10:17.307734 16248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:10:17.308053 16248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:10:17.308277 16248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:10:17.316289 16248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:10:17.316417 16248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:10:17.316625 16248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:10:17.327133 16248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:10:17.328547 16248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:10:17.330003 16248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:10:17.345083 16254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:17.345130 16253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:17.345367 16256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:10:17.346060 16248 server_base.cc:1034] running on GCE node
I20260619 01:10:17.346766 16248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:10:17.347751 16248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:10:17.348971 16248 hybrid_clock.cc:648] HybridClock initialized: now 1781831417348918 us; error 40 us; skew 500 ppm
I20260619 01:10:17.349416 16248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:10:17 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:10:17.352902 16248 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:10:17.354842 16248 webserver.cc:460] Webserver started at http://127.0.97.3:40643/ using document root <none> and password file <none>
I20260619 01:10:17.355635 16248 fs_manager.cc:359] Metadata directory not provided
I20260619 01:10:17.355782 16248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:10:17.356132 16248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:10:17.359406 16248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "a0a972f4609f4ea0a0e835293b90f45c"
format_stamp: "Formatted at 2026-06-19 01:10:17 on dist-test-slave-5p8w"
server_key: "7200ac25c846d1936b691495d08d5dd9"
server_key_iv: "4a6ff560983719a4af0230ed5dd4b200"
server_key_version: "encryptionkey@0"
I20260619 01:10:17.360252 16248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a0a972f4609f4ea0a0e835293b90f45c"
format_stamp: "Formatted at 2026-06-19 01:10:17 on dist-test-slave-5p8w"
server_key: "7200ac25c846d1936b691495d08d5dd9"
server_key_iv: "4a6ff560983719a4af0230ed5dd4b200"
server_key_version: "encryptionkey@0"
I20260619 01:10:17.366017 16248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 01:10:17.369048 16108 heartbeater.cc:503] Master 127.0.97.62:38821 was elected leader, sending a full tablet report...
I20260619 01:10:17.370131 16263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:17.371912 16248 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:10:17.372081 16248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "a0a972f4609f4ea0a0e835293b90f45c"
format_stamp: "Formatted at 2026-06-19 01:10:17 on dist-test-slave-5p8w"
server_key: "7200ac25c846d1936b691495d08d5dd9"
server_key_iv: "4a6ff560983719a4af0230ed5dd4b200"
server_key_version: "encryptionkey@0"
I20260619 01:10:17.372267 16248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:10:17.401345 16248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:10:17.405951 16248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:10:17.406224 16248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:10:17.407236 16248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:10:17.408870 16248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:10:17.408948 16248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:17.409077 16248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:10:17.409142 16248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:17.444643 16248 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42087
I20260619 01:10:17.444725 16375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42087 every 8 connection(s)
I20260619 01:10:17.446002 16248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:10:17.449054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 16248
I20260619 01:10:17.449276 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:10:17.449895 388 external_mini_cluster.cc:1428] Setting key 582a860fe26cfbb941433ebffaa777f3
Jun 19 01:10:17 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:10:17.467545 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.448760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39695 (local address 127.0.97.62:38821)
0619 01:10:17.449270 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:17.449277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:17.450720 (+ 1443us) server_negotiation.cc:407] Connection header received
0619 01:10:17.452771 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:17.452775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:17.452848 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:17.452936 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:17.454637 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.455410 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.456968 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.457490 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.457592 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.461230 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:17.461236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:17.461239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:17.461277 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:17.463994 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:17.465023 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:17.465032 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:17.465036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:17.465177 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:17.465787 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:17.465791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:17.465793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:17.466190 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:17.466353 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:17.467071 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 01:10:17.467262 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":235,"threads_started":1}
I20260619 01:10:17.468555 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.449398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:39695)
0619 01:10:17.450270 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:17.450327 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:17.452460 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:17.453238 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:17.453249 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:17.453871 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:10:17.454462 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.454480 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.455547 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.455552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.456831 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.456848 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.457693 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.457697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.457907 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.458630 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:10:17.458655 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:10:17.460989 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:10:17.464186 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:17.464196 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:17.464219 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:17.464866 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:17.465338 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:17.465367 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:17.465372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:17.465668 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:17.466341 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:10:17.466348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:10:17.466711 (+ 363us) client_negotiation.cc:769] Sending connection context
0619 01:10:17.466959 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 01:10:17.467389 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":305,"threads_started":1}
I20260619 01:10:17.470165 16376 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260619 01:10:17.470481 16376 heartbeater.cc:464] Registering TS with master...
I20260619 01:10:17.471203 16376 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260619 01:10:17.472750 15919 ts_manager.cc:194] Registered new tserver with Master: a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087)
I20260619 01:10:17.473759 15919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39695
I20260619 01:10:17.478619 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:10:17.496206 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.483267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.97.62:38821)
0619 01:10:17.483517 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:17.483526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:17.483915 (+ 389us) server_negotiation.cc:407] Connection header received
0619 01:10:17.484061 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:17.484065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:17.484142 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:17.484226 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:17.485824 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.486552 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.487977 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.488397 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.488503 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.491072 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:17.491077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:17.491080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:17.491116 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:17.493605 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:17.494353 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:17.494356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:17.494359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:17.494477 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:17.494810 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:17.494813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:17.494815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:17.495171 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:17.495374 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:17.495716 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 01:10:17.495902 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 01:10:17.500514 15919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:10:17.507617 15919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:10:17.535077 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.520947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35715 (local address 127.0.0.1:33128)
0619 01:10:17.522388 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:17.522453 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:17.522683 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:17.524039 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:17.524051 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:17.524093 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:10:17.524596 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.524613 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.527270 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.527279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.530674 (+ 3395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.530702 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.533926 (+ 3224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.533935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.534220 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.534289 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 01:10:17.534429 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 01:10:17.534662 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"thread_start_us":924,"threads_started":1}
I20260619 01:10:17.535095 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.521157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.0.97.2:35715)
0619 01:10:17.521838 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:17.521849 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:17.522917 (+ 1068us) server_negotiation.cc:407] Connection header received
0619 01:10:17.523061 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:17.523072 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:17.523416 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:17.523637 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:10:17.524834 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.527047 (+ 2213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.531000 (+ 3953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.533208 (+ 2208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.533498 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.533768 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:17.534582 (+ 814us) server_negotiation.cc:299] Negotiation successful
0619 01:10:17.534772 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":387,"threads_started":1}
I20260619 01:10:17.540666 16177 tablet_service.cc:1467] Processing CreateTablet for tablet e7dc6be21fcf453fab3699672cb80cf0 (DEFAULT_TABLE table=test-table [id=05b203ef8059474aa5888125a9570e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:10:17.542295 16177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7dc6be21fcf453fab3699672cb80cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:10:17.543915 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.522819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.0.1:54016)
0619 01:10:17.535745 (+ 12926us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:17.535765 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:17.535875 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:17.537062 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:17.537066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:17.537096 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:10:17.537246 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.537257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.539224 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.539230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.541184 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.541200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.543069 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.543075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.543269 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.543306 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 01:10:17.543408 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 01:10:17.543596 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12778,"thread_start_us":13324,"threads_started":1}
I20260619 01:10:17.543965 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.523488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.0.97.1:39611)
0619 01:10:17.536009 (+ 12521us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:17.536020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:17.536060 (+ 40us) server_negotiation.cc:407] Connection header received
0619 01:10:17.536178 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:17.536186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:17.536535 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:17.536772 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:10:17.537426 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.539054 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.541441 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.542891 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.543056 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.543305 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:17.543478 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 01:10:17.543655 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12317,"thread_start_us":12130,"threads_started":1}
I20260619 01:10:17.545619 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.521168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42087 (local address 127.0.0.1:39656)
0619 01:10:17.533917 (+ 12749us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:17.533945 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:17.534376 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:17.537511 (+ 3135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:17.537515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:17.537542 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:10:17.537893 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.537909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.540337 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.540343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.542499 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.542514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.544776 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.544783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.544977 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.545011 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 01:10:17.545080 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 01:10:17.545214 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12576,"thread_start_us":12930,"threads_started":1}
I20260619 01:10:17.546363 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.521919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.0.97.3:42087)
0619 01:10:17.536827 (+ 14908us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:17.536835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:17.536873 (+ 38us) server_negotiation.cc:407] Connection header received
0619 01:10:17.536985 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:17.536994 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:17.537259 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:17.538128 (+ 869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:10:17.538494 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.540176 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.542679 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.544539 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.545495 (+ 956us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.545739 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:17.545820 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 01:10:17.546009 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14729,"thread_start_us":14568,"threads_started":1}
I20260619 01:10:17.548460 16043 tablet_service.cc:1467] Processing CreateTablet for tablet e7dc6be21fcf453fab3699672cb80cf0 (DEFAULT_TABLE table=test-table [id=05b203ef8059474aa5888125a9570e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:10:17.550215 16043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7dc6be21fcf453fab3699672cb80cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:10:17.550489 16311 tablet_service.cc:1467] Processing CreateTablet for tablet e7dc6be21fcf453fab3699672cb80cf0 (DEFAULT_TABLE table=test-table [id=05b203ef8059474aa5888125a9570e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:10:17.552023 16311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7dc6be21fcf453fab3699672cb80cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:10:17.555809 16395 tablet_bootstrap.cc:492] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: Bootstrap starting.
I20260619 01:10:17.559046 16395 tablet_bootstrap.cc:654] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: Neither blocks nor log segments found. Creating new log.
I20260619 01:10:17.560273 16395 log.cc:824] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: Log is configured to *not* fsync() on all Append() calls
I20260619 01:10:17.563773 16395 tablet_bootstrap.cc:492] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: No bootstrap required, opened a new log
I20260619 01:10:17.564082 16395 ts_tablet_manager.cc:1397] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260619 01:10:17.565642 16397 tablet_bootstrap.cc:492] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: Bootstrap starting.
I20260619 01:10:17.567848 16398 tablet_bootstrap.cc:492] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: Bootstrap starting.
I20260619 01:10:17.570619 16397 tablet_bootstrap.cc:654] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: Neither blocks nor log segments found. Creating new log.
I20260619 01:10:17.570946 16398 tablet_bootstrap.cc:654] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: Neither blocks nor log segments found. Creating new log.
I20260619 01:10:17.572041 16397 log.cc:824] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: Log is configured to *not* fsync() on all Append() calls
I20260619 01:10:17.572103 16398 log.cc:824] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: Log is configured to *not* fsync() on all Append() calls
I20260619 01:10:17.575311 16395 raft_consensus.cc:348] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } }
I20260619 01:10:17.575780 16395 raft_consensus.cc:374] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:10:17.575896 16397 tablet_bootstrap.cc:492] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: No bootstrap required, opened a new log
I20260619 01:10:17.575974 16395 raft_consensus.cc:729] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79e1bb3913541228217340ebb478f76, State: Initialized, Role: FOLLOWER
I20260619 01:10:17.576135 16398 tablet_bootstrap.cc:492] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: No bootstrap required, opened a new log
I20260619 01:10:17.576196 16397 ts_tablet_manager.cc:1397] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 01:10:17.576400 16398 ts_tablet_manager.cc:1397] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 01:10:17.576689 16395 consensus_queue.cc:260] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } }
I20260619 01:10:17.584115 16242 heartbeater.cc:503] Master 127.0.97.62:38821 was elected leader, sending a full tablet report...
I20260619 01:10:17.585278 16395 ts_tablet_manager.cc:1428] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: Time spent starting tablet: real 0.021s user 0.014s sys 0.006s
I20260619 01:10:17.587704 16397 raft_consensus.cc:348] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } }
I20260619 01:10:17.587707 16398 raft_consensus.cc:348] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } }
I20260619 01:10:17.588215 16398 raft_consensus.cc:374] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:10:17.588219 16397 raft_consensus.cc:374] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:10:17.588328 16397 raft_consensus.cc:729] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a992b7f6c9843ecbcfcc095cff9d893, State: Initialized, Role: FOLLOWER
I20260619 01:10:17.588416 16398 raft_consensus.cc:729] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a972f4609f4ea0a0e835293b90f45c, State: Initialized, Role: FOLLOWER
I20260619 01:10:17.589114 16398 consensus_queue.cc:260] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } }
I20260619 01:10:17.589114 16397 consensus_queue.cc:260] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } }
I20260619 01:10:17.591183 16376 heartbeater.cc:503] Master 127.0.97.62:38821 was elected leader, sending a full tablet report...
I20260619 01:10:17.591419 16397 ts_tablet_manager.cc:1428] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260619 01:10:17.591646 16398 ts_tablet_manager.cc:1428] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260619 01:10:17.627724 16243 tablet.cc:2369] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:10:17.697602 16377 tablet.cc:2369] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:10:17.837081 16109 tablet.cc:2369] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:10:17.957490 16402 raft_consensus.cc:482] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:10:17.957815 16402 raft_consensus.cc:504] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } }
I20260619 01:10:17.959584 16402 leader_election.cc:288] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715), 1a992b7f6c9843ecbcfcc095cff9d893 (127.0.97.1:39611)
I20260619 01:10:17.966064 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.959837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35715 (local address 127.0.97.3:59957)
0619 01:10:17.960087 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:17.960105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:17.960267 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:17.960763 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:17.960766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:17.960790 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:10:17.960999 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.961011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.962336 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.962342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.963883 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.963896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.965239 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.965246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.965481 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.965520 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 01:10:17.965624 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 01:10:17.965784 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":23}
I20260619 01:10:17.966086 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.960107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59957 (local address 127.0.97.2:35715)
0619 01:10:17.960334 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:17.960341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:17.960362 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:10:17.960445 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:17.960450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:17.960549 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:17.960652 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:10:17.961160 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.962187 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.964055 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.965072 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.965178 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.965252 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:17.965656 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 01:10:17.965817 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 01:10:17.966686 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.960273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.97.3:51393)
0619 01:10:17.960851 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:17.960865 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:17.960970 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:17.961318 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:17.961323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:17.961348 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:10:17.961684 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.961697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.962878 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.962882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.964468 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:17.964486 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.965821 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.965827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:17.966020 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.966055 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 01:10:17.966159 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 01:10:17.966296 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":3,"thread_start_us":273,"threads_started":1}
I20260619 01:10:17.966909 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:17.960459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51393 (local address 127.0.97.1:39611)
0619 01:10:17.960677 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:17.960682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:17.960966 (+ 284us) server_negotiation.cc:407] Connection header received
0619 01:10:17.961079 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:17.961087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:17.961173 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:17.961240 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:10:17.961836 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.962706 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.964628 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:17.965618 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:17.965737 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:17.965829 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:17.966501 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 01:10:17.966640 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 01:10:17.967321 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7dc6be21fcf453fab3699672cb80cf0" candidate_uuid: "a0a972f4609f4ea0a0e835293b90f45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79e1bb3913541228217340ebb478f76" is_pre_election: true
I20260619 01:10:17.967882 16197 raft_consensus.cc:2454] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0a972f4609f4ea0a0e835293b90f45c in term 0.
I20260619 01:10:17.967886 16063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7dc6be21fcf453fab3699672cb80cf0" candidate_uuid: "a0a972f4609f4ea0a0e835293b90f45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" is_pre_election: true
I20260619 01:10:17.968487 16063 raft_consensus.cc:2454] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0a972f4609f4ea0a0e835293b90f45c in term 0.
I20260619 01:10:17.968614 16267 leader_election.cc:302] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a972f4609f4ea0a0e835293b90f45c, c79e1bb3913541228217340ebb478f76; no voters:
I20260619 01:10:17.969174 16402 raft_consensus.cc:2781] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:10:17.969381 16402 raft_consensus.cc:482] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:10:17.969506 16402 raft_consensus.cc:3037] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:10:17.971755 16402 raft_consensus.cc:504] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } }
I20260619 01:10:17.972373 16402 leader_election.cc:288] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [CANDIDATE]: Term 1 election: Requested vote from peers c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715), 1a992b7f6c9843ecbcfcc095cff9d893 (127.0.97.1:39611)
I20260619 01:10:17.972983 16063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7dc6be21fcf453fab3699672cb80cf0" candidate_uuid: "a0a972f4609f4ea0a0e835293b90f45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a992b7f6c9843ecbcfcc095cff9d893"
I20260619 01:10:17.972996 16197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7dc6be21fcf453fab3699672cb80cf0" candidate_uuid: "a0a972f4609f4ea0a0e835293b90f45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79e1bb3913541228217340ebb478f76"
I20260619 01:10:17.973212 16063 raft_consensus.cc:3037] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:10:17.973210 16197 raft_consensus.cc:3037] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:10:17.976565 16063 raft_consensus.cc:2454] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a972f4609f4ea0a0e835293b90f45c in term 1.
I20260619 01:10:17.976565 16197 raft_consensus.cc:2454] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a972f4609f4ea0a0e835293b90f45c in term 1.
I20260619 01:10:17.976992 16267 leader_election.cc:302] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a972f4609f4ea0a0e835293b90f45c, c79e1bb3913541228217340ebb478f76; no voters:
I20260619 01:10:17.977500 16402 raft_consensus.cc:2781] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:10:17.977962 16402 raft_consensus.cc:686] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 1 LEADER]: Becoming Leader. State: Replica: a0a972f4609f4ea0a0e835293b90f45c, State: Running, Role: LEADER
I20260619 01:10:17.978516 16402 consensus_queue.cc:237] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } }
I20260619 01:10:17.983237 15919 catalog_manager.cc:5495] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a0a972f4609f4ea0a0e835293b90f45c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } health_report { overall_health: UNKNOWN } } }
I20260619 01:10:18.006412 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:18.000229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.0.97.3:42087)
0619 01:10:18.000458 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:18.000465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:18.000832 (+ 367us) server_negotiation.cc:407] Connection header received
0619 01:10:18.001092 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:18.001097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:18.001179 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:18.001389 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:10:18.001851 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:18.002597 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:18.004098 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:18.004513 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:18.004650 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:18.005181 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:10:18.005797 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:10:18.005971 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:18.006054 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 01:10:18.006139 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:10:18.014842 16063 raft_consensus.cc:1261] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Refusing update from remote peer a0a972f4609f4ea0a0e835293b90f45c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:10:18.014891 16197 raft_consensus.cc:1261] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76 [term 1 FOLLOWER]: Refusing update from remote peer a0a972f4609f4ea0a0e835293b90f45c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:10:18.015915 16405 consensus_queue.cc:1035] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:10:18.016336 16402 consensus_queue.cc:1035] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:10:18.052280 15918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:10:18.052597 15918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:10:18.056665 15918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Sending DeleteTablet for 3 replicas of tablet e7dc6be21fcf453fab3699672cb80cf0
I20260619 01:10:18.058077 16311 tablet_service.cc:1514] Processing DeleteTablet for tablet e7dc6be21fcf453fab3699672cb80cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:10:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39656
I20260619 01:10:18.058465 16043 tablet_service.cc:1514] Processing DeleteTablet for tablet e7dc6be21fcf453fab3699672cb80cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:10:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54016
I20260619 01:10:18.058487 16177 tablet_service.cc:1514] Processing DeleteTablet for tablet e7dc6be21fcf453fab3699672cb80cf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:10:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33128
I20260619 01:10:18.059469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 15980
I20260619 01:10:18.070709 16422 tablet_replica.cc:330] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: stopping tablet replica
I20260619 01:10:18.071091 16420 tablet_replica.cc:330] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: stopping tablet replica
I20260619 01:10:18.071432 16422 raft_consensus.cc:2229] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:10:18.071604 16420 raft_consensus.cc:2229] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:10:18.072434 16420 raft_consensus.cc:2258] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:10:18.073359 16422 raft_consensus.cc:2258] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:10:18.074290 16421 tablet_replica.cc:330] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: stopping tablet replica
I20260619 01:10:18.074915 16421 raft_consensus.cc:2229] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:10:18.075515 16421 raft_consensus.cc:2258] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:10:18.075628 16422 ts_tablet_manager.cc:1905] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:10:18.077013 16420 ts_tablet_manager.cc:1905] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:10:18.077592 16421 ts_tablet_manager.cc:1905] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 01:10:18 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:10:18.085197 16420 ts_tablet_manager.cc:1918] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:10:18.085453 16420 log.cc:1196] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/e7dc6be21fcf453fab3699672cb80cf0
I20260619 01:10:18.086422 16420 ts_tablet_manager.cc:1939] T e7dc6be21fcf453fab3699672cb80cf0 P a0a972f4609f4ea0a0e835293b90f45c: Deleting consensus metadata
I20260619 01:10:18.088677 15903 catalog_manager.cc:4841] TS a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087): tablet e7dc6be21fcf453fab3699672cb80cf0 (table test-table [id=05b203ef8059474aa5888125a9570e89]) successfully deleted
I20260619 01:10:18.090581 16422 ts_tablet_manager.cc:1918] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:10:18.090883 16422 log.cc:1196] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/e7dc6be21fcf453fab3699672cb80cf0
I20260619 01:10:18.091517 16422 ts_tablet_manager.cc:1939] T e7dc6be21fcf453fab3699672cb80cf0 P c79e1bb3913541228217340ebb478f76: Deleting consensus metadata
I20260619 01:10:18.091552 16421 ts_tablet_manager.cc:1918] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:10:18.091765 16421 log.cc:1196] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/e7dc6be21fcf453fab3699672cb80cf0
I20260619 01:10:18.092337 16421 ts_tablet_manager.cc:1939] T e7dc6be21fcf453fab3699672cb80cf0 P 1a992b7f6c9843ecbcfcc095cff9d893: Deleting consensus metadata
I20260619 01:10:18.093397 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:18.061066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.0.97.1:39611)
0619 01:10:18.061309 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:18.061318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:18.069016 (+ 7698us) server_negotiation.cc:407] Connection header received
0619 01:10:18.070208 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:18.070217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:18.070371 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:18.070449 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:18.074495 (+ 4046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:18.075690 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:18.077910 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:18.078550 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:18.078661 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:18.084884 (+ 6223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:18.084902 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:18.084924 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:18.084977 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:18.089338 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:18.090378 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:18.090389 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:18.090393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:18.090584 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:18.091083 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:18.091088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:18.091091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:18.091622 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:18.091950 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:18.092687 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 01:10:18.092921 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:10:18.094020 15906 catalog_manager.cc:4841] TS c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715): tablet e7dc6be21fcf453fab3699672cb80cf0 (table test-table [id=05b203ef8059474aa5888125a9570e89]) successfully deleted
I20260619 01:10:18.094625 16103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:10:18.094959 15906 catalog_manager.cc:4841] TS 1a992b7f6c9843ecbcfcc095cff9d893 (127.0.97.1:39611): tablet e7dc6be21fcf453fab3699672cb80cf0 (table test-table [id=05b203ef8059474aa5888125a9570e89]) successfully deleted
I20260619 01:10:18.188670 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 15980
I20260619 01:10:18.212930 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 16114
Jun 19 01:10:18 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:10:18.229092 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:18.213839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.0.97.2:35715)
0619 01:10:18.214048 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:18.214054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:18.214077 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:10:18.214151 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:18.214156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:18.214218 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:18.214322 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:18.215711 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:18.216425 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:18.217972 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:18.218373 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:18.218522 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:18.223134 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:18.223151 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:18.223167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:18.223213 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:18.226229 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:18.226941 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:18.226950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:18.226953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:18.227077 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:18.227478 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:18.227483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:18.227488 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:18.227867 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:18.228159 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:18.228517 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 01:10:18.228739 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:10:18.229904 16237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:10:18.321938 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 16114
I20260619 01:10:18.345999 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 16248
Jun 19 01:10:18 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:10:18.362766 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:18.346793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.0.97.3:42087)
0619 01:10:18.347021 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:18.347027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:18.347053 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:10:18.347154 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:18.347159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:18.347251 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:18.347359 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:18.348693 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:18.349611 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:18.350938 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:18.351424 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:18.351567 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:18.356485 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:18.356501 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:18.356518 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:18.356560 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:18.359762 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:18.360512 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:18.360520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:18.360522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:18.360677 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:18.361084 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:18.361090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:18.361099 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:18.361497 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:18.361825 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:18.362147 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 01:10:18.362387 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 01:10:18.363606 16371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:10:18.487255 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 16248
I20260619 01:10:18.504457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 15887
I20260619 01:10:18.505676 15949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:10:18.617583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 15887
I20260619 01:10:38.640339 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38821
--webserver_interface=127.0.97.62
--webserver_port=36171
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:10:38.994666 16428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:10:38.994992 16428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:10:38.995169 16428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:10:39.003139 16428 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 01:10:39.003270 16428 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 01:10:39.003356 16428 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 01:10:39.003443 16428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:10:39.003551 16428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:10:39.003672 16428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:10:39.003742 16428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:10:39.003832 16428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:10:39.014513 16428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:10:39.015992 16428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:10:39.017503 16428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:10:39.042433 16434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:39.042488 16435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:39.042784 16437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:10:39.042575 16428 server_base.cc:1034] running on GCE node
I20260619 01:10:39.044044 16428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:10:39.045866 16428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:10:39.047173 16428 hybrid_clock.cc:648] HybridClock initialized: now 1781831439047096 us; error 49 us; skew 500 ppm
I20260619 01:10:39.047616 16428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:10:39 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:10:39.057443 16428 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:10:39.059361 16428 webserver.cc:460] Webserver started at http://127.0.97.62:36171/ using document root <none> and password file <none>
I20260619 01:10:39.060247 16428 fs_manager.cc:359] Metadata directory not provided
I20260619 01:10:39.060405 16428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:10:39.067231 16428 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 01:10:39.070634 16444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:39.072270 16428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:10:39.072523 16428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "23df21a0905645d2a99a3b7e8c348ddb"
format_stamp: "Formatted at 2026-06-19 01:10:15 on dist-test-slave-5p8w"
server_key: "00ac694168a0d0e7afea4f52e4726776"
server_key_iv: "81b899a32bfb6b4bf52926a776567f4a"
server_key_version: "encryptionkey@0"
I20260619 01:10:39.073858 16428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:10:39.090090 16428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:10:39.095274 16428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:10:39.095611 16428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:10:39.116554 16428 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38821
I20260619 01:10:39.116575 16495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38821 every 8 connection(s)
I20260619 01:10:39.118419 16428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:10:39.126283 16496 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:10:39.126767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 16428
I20260619 01:10:39.127594 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39611
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38821
--webserver_port=38783
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:10:39.131979 16496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Bootstrap starting.
I20260619 01:10:39.166235 16496 log.cc:824] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Log is configured to *not* fsync() on all Append() calls
I20260619 01:10:39.185920 16496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:10:39.186738 16496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Bootstrap complete.
I20260619 01:10:39.202872 16496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:10:39.204330 16496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23df21a0905645d2a99a3b7e8c348ddb, State: Initialized, Role: FOLLOWER
I20260619 01:10:39.205240 16496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:10:39.205631 16496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:10:39.205793 16496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:10:39.206008 16496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:10:39.214058 16496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:10:39.214905 16496 leader_election.cc:302] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23df21a0905645d2a99a3b7e8c348ddb; no voters:
I20260619 01:10:39.215698 16496 leader_election.cc:288] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 01:10:39.215898 16500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 2 FOLLOWER]: Leader election won for term 2
I20260619 01:10:39.217377 16500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 2 LEADER]: Becoming Leader. State: Replica: 23df21a0905645d2a99a3b7e8c348ddb, State: Running, Role: LEADER
I20260619 01:10:39.218086 16500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:10:39.218258 16496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:10:39.220937 16502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23df21a0905645d2a99a3b7e8c348ddb. Latest consensus state: current_term: 2 leader_uuid: "23df21a0905645d2a99a3b7e8c348ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } } }
I20260619 01:10:39.220858 16501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "23df21a0905645d2a99a3b7e8c348ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } } }
I20260619 01:10:39.221354 16501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: This master's current role is: LEADER
I20260619 01:10:39.221351 16502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: This master's current role is: LEADER
I20260619 01:10:39.232564 16506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:10:39.243360 16506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:10:39.245086 16506 catalog_manager.cc:1259] Loaded cluster ID: c5b6de766ad44ac7b2518e413ec8af24
I20260619 01:10:39.245325 16506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:10:39.249788 16506 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:10:39.251673 16506 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Loaded TSK: 0
I20260619 01:10:39.262822 16517 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 01:10:39.289657 16506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Generated new TSK 1
I20260619 01:10:39.290141 16506 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 01:10:39.546286 16498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:10:39.546640 16498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:10:39.546816 16498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:10:39.554589 16498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:10:39.554761 16498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:10:39.554939 16498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:10:39.565378 16498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38783
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:10:39.566830 16498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:10:39.568279 16498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:10:39.583832 16524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:39.583811 16523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:39.584827 16526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:10:39.585858 16498 server_base.cc:1034] running on GCE node
I20260619 01:10:39.586601 16498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:10:39.587705 16498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:10:39.588980 16498 hybrid_clock.cc:648] HybridClock initialized: now 1781831439588913 us; error 64 us; skew 500 ppm
I20260619 01:10:39.589432 16498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:10:39 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:10:39.593161 16498 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:10:39.595072 16498 webserver.cc:460] Webserver started at http://127.0.97.1:38783/ using document root <none> and password file <none>
I20260619 01:10:39.595852 16498 fs_manager.cc:359] Metadata directory not provided
I20260619 01:10:39.596014 16498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:10:39.602133 16498 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 01:10:39.605234 16533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:39.606885 16498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:10:39.607079 16498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "1a992b7f6c9843ecbcfcc095cff9d893"
format_stamp: "Formatted at 2026-06-19 01:10:16 on dist-test-slave-5p8w"
server_key: "2926ba4e97fba9e014098dc4866f310e"
server_key_iv: "c4f81fd8745c188387eedcc6cd6073cc"
server_key_version: "encryptionkey@0"
I20260619 01:10:39.608098 16498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:10:39.631055 16498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:10:39.635684 16498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:10:39.635995 16498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:10:39.637049 16498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:10:39.638551 16498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:10:39.638644 16498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:39.638823 16498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:10:39.638928 16498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:39.673775 16498 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39611
I20260619 01:10:39.673906 16645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39611 every 8 connection(s)
I20260619 01:10:39.675316 16498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:10:39.684602 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 16498
I20260619 01:10:39.685518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35715
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38821
--webserver_port=34625
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:10:39 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:10:39.703917 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:39.678526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45865 (local address 127.0.97.62:38821)
0619 01:10:39.679089 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:39.679103 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:39.679981 (+ 878us) server_negotiation.cc:407] Connection header received
0619 01:10:39.682404 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:39.682419 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:39.682799 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:39.683244 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:39.685500 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:39.686927 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:39.689631 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:39.690347 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:39.690504 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:39.695737 (+ 5233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:39.695747 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:39.695765 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:39.695822 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:39.698957 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:39.700075 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:39.700082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:39.700089 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:39.700267 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:39.700808 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:39.700811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:39.700813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:39.701398 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:39.701553 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:39.702565 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0619 01:10:39.702912 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
I20260619 01:10:39.704416 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:39.678892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:45865)
0619 01:10:39.679761 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:39.679831 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:39.681768 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:39.683408 (+ 1640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:39.683421 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:39.684245 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:10:39.685152 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:39.685186 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:39.687133 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:39.687142 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:39.689436 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:39.689458 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:39.690633 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:39.690642 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:39.690990 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:39.691826 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:10:39.691870 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:10:39.695335 (+ 3465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:10:39.699162 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:39.699175 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:39.699197 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:39.699918 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:39.700393 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:39.700401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:39.700408 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:39.700689 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:39.701535 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:10:39.701546 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:10:39.702030 (+ 484us) client_negotiation.cc:769] Sending connection context
0619 01:10:39.702367 (+ 337us) client_negotiation.cc:240] Negotiation successful
0619 01:10:39.703021 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":316,"threads_started":1}
I20260619 01:10:39.708089 16646 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260619 01:10:39.708505 16646 heartbeater.cc:464] Registering TS with master...
I20260619 01:10:39.709683 16646 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260619 01:10:39.712872 16461 ts_manager.cc:194] Registered new tserver with Master: 1a992b7f6c9843ecbcfcc095cff9d893 (127.0.97.1:39611)
I20260619 01:10:39.715219 16461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45865
W20260619 01:10:40.043769 16650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:10:40.044103 16650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:10:40.044205 16650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:10:40.051791 16650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:10:40.051939 16650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:10:40.052057 16650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:10:40.062460 16650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34625
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:10:40.063903 16650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:10:40.065261 16650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:10:40.080438 16655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:40.080488 16656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:40.080834 16658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:10:40.081615 16650 server_base.cc:1034] running on GCE node
I20260619 01:10:40.082369 16650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:10:40.083463 16650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:10:40.084703 16650 hybrid_clock.cc:648] HybridClock initialized: now 1781831440084641 us; error 61 us; skew 500 ppm
I20260619 01:10:40.085099 16650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:10:40 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:10:40.088770 16650 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:10:40.090622 16650 webserver.cc:460] Webserver started at http://127.0.97.2:34625/ using document root <none> and password file <none>
I20260619 01:10:40.091450 16650 fs_manager.cc:359] Metadata directory not provided
I20260619 01:10:40.091630 16650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:10:40.097878 16650 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 01:10:40.101055 16665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:40.102849 16650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:10:40.103097 16650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "c79e1bb3913541228217340ebb478f76"
format_stamp: "Formatted at 2026-06-19 01:10:16 on dist-test-slave-5p8w"
server_key: "189b9452a1385f7cc7ea0e9367ecece7"
server_key_iv: "a8db51466d4918e3d5883e9d5a302f89"
server_key_version: "encryptionkey@0"
I20260619 01:10:40.104290 16650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:10:40.130985 16650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:10:40.136641 16650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:10:40.136956 16650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:10:40.138105 16650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:10:40.139523 16650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:10:40.139657 16650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:40.139799 16650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:10:40.139896 16650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:40.173493 16650 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35715
I20260619 01:10:40.173557 16777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35715 every 8 connection(s)
I20260619 01:10:40.175096 16650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:10:40.185256 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 16650
I20260619 01:10:40.186044 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42087
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38821
--webserver_port=40643
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:10:40 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:10:40.198251 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.178056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55615 (local address 127.0.97.62:38821)
0619 01:10:40.178252 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.178257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.179586 (+ 1329us) server_negotiation.cc:407] Connection header received
0619 01:10:40.181606 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.181611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.181699 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.181794 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:40.183440 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.184203 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.185988 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.186792 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.186961 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.191040 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:40.191050 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:40.191058 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:40.191114 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:40.194597 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:40.195686 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:40.195691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:40.195694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:40.195845 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:40.196345 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:40.196349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:40.196351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:40.196647 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:40.196822 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:40.197624 (+ 802us) server_negotiation.cc:299] Negotiation successful
0619 01:10:40.197895 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:10:40.199357 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.178611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:55615)
0619 01:10:40.179417 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:40.179463 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:40.181261 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:40.182037 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:40.182046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:40.182644 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:10:40.183245 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.183264 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.184366 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.184370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.185815 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.185828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.187061 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.187067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.187284 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.188021 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:10:40.188049 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:10:40.190696 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:10:40.194826 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:40.194838 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:40.194861 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:40.195508 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:40.195974 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:40.195981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:40.195989 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:40.196196 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:40.196789 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:10:40.196798 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:10:40.197180 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 01:10:40.197530 (+ 350us) client_negotiation.cc:240] Negotiation successful
0619 01:10:40.198118 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":314,"threads_started":1}
I20260619 01:10:40.200933 16778 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260619 01:10:40.201313 16778 heartbeater.cc:464] Registering TS with master...
I20260619 01:10:40.202260 16778 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260619 01:10:40.203974 16461 ts_manager.cc:194] Registered new tserver with Master: c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715)
I20260619 01:10:40.205176 16461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55615
W20260619 01:10:40.537593 16781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:10:40.537899 16781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:10:40.538110 16781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:10:40.545537 16781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:10:40.545661 16781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:10:40.545863 16781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:10:40.555974 16781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40643
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:10:40.557427 16781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:10:40.559022 16781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:10:40.573680 16787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:40.573598 16786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:10:40.573701 16789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:10:40.574618 16781 server_base.cc:1034] running on GCE node
I20260619 01:10:40.575341 16781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:10:40.576380 16781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:10:40.577585 16781 hybrid_clock.cc:648] HybridClock initialized: now 1781831440577509 us; error 70 us; skew 500 ppm
I20260619 01:10:40.577955 16781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:10:40 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:10:40.581714 16781 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:10:40.583559 16781 webserver.cc:460] Webserver started at http://127.0.97.3:40643/ using document root <none> and password file <none>
I20260619 01:10:40.584406 16781 fs_manager.cc:359] Metadata directory not provided
I20260619 01:10:40.584584 16781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:10:40.590553 16781 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:10:40.593669 16796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:40.595455 16781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:10:40.595712 16781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "a0a972f4609f4ea0a0e835293b90f45c"
format_stamp: "Formatted at 2026-06-19 01:10:17 on dist-test-slave-5p8w"
server_key: "7200ac25c846d1936b691495d08d5dd9"
server_key_iv: "4a6ff560983719a4af0230ed5dd4b200"
server_key_version: "encryptionkey@0"
I20260619 01:10:40.596937 16781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:10:40.625677 16781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:10:40.630473 16781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:10:40.630786 16781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:10:40.631915 16781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:10:40.633476 16781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:10:40.633607 16781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:40.633744 16781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:10:40.633833 16781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:10:40.668982 16781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42087
I20260619 01:10:40.669075 16908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42087 every 8 connection(s)
I20260619 01:10:40.670552 16781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:10:40.675130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 16781
Jun 19 01:10:40 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:10:40.699688 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.673660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46771 (local address 127.0.97.62:38821)
0619 01:10:40.673873 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.673879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.676214 (+ 2335us) server_negotiation.cc:407] Connection header received
0619 01:10:40.680198 (+ 3984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.680209 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.680359 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.680456 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:40.683974 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.685654 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.687583 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.688080 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.688237 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.692460 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:40.692466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:40.692471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:40.692527 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:40.695550 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:40.696713 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:40.696720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:40.696725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:40.696860 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:40.697408 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:40.697412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:40.697414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:40.697876 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:40.698046 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:40.698991 (+ 945us) server_negotiation.cc:299] Negotiation successful
0619 01:10:40.699259 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":77}
I20260619 01:10:40.700564 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.674110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:46771)
0619 01:10:40.676037 (+ 1927us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:40.676089 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:40.679667 (+ 3578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:40.681259 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:40.681277 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:40.682449 (+ 1172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:10:40.683590 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.683627 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.685936 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.685943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.687379 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.687394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.688356 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.688364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.688639 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.689443 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:10:40.689468 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:10:40.692135 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:10:40.695762 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:40.695776 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:40.695802 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:40.696568 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:40.696982 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:10:40.696986 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:10:40.696988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:10:40.697202 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:10:40.698029 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:10:40.698037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:10:40.698481 (+ 444us) client_negotiation.cc:769] Sending connection context
0619 01:10:40.698808 (+ 327us) client_negotiation.cc:240] Negotiation successful
0619 01:10:40.699286 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1624,"thread_start_us":359,"threads_started":1}
I20260619 01:10:40.702420 16909 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38821
I20260619 01:10:40.702819 16909 heartbeater.cc:464] Registering TS with master...
I20260619 01:10:40.703590 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.676267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.0.97.62:38821)
0619 01:10:40.685898 (+ 9631us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.685904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.687671 (+ 1767us) server_negotiation.cc:407] Connection header received
0619 01:10:40.687729 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.687733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.687800 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.687942 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:40.689724 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.690518 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.692424 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.693199 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.693397 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.696547 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:40.696552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:40.696555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:40.696590 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:40.700152 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:40.701207 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:40.701214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:40.701223 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:40.701419 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:40.701945 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:40.701951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:40.701954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:40.702376 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:40.702667 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:40.703036 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 01:10:40.703224 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":9447,"thread_start_us":9282,"threads_started":1}
I20260619 01:10:40.703835 16909 heartbeater.cc:511] Master 127.0.97.62:38821 requested a full tablet report, sending...
I20260619 01:10:40.705543 16461 ts_manager.cc:194] Registered new tserver with Master: a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087)
I20260619 01:10:40.706463 16461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46771
I20260619 01:10:40.706662 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:10:40.713948 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.707665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46774 (local address 127.0.97.62:38821)
0619 01:10:40.707947 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.707953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.708368 (+ 415us) server_negotiation.cc:407] Connection header received
0619 01:10:40.708550 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.708556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.708833 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.709014 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:10:40.709476 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.710374 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.711950 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.712392 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.712585 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.713091 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:10:40.713408 (+ 317us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 01:10:40.713721 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46774: token expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 01:10:40.714104 16913 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46774: token expired
W20260619 01:10:40.714679 16382 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:38821)
I20260619 01:10:40.719488 16646 heartbeater.cc:503] Master 127.0.97.62:38821 was elected leader, sending a full tablet report...
I20260619 01:10:40.728721 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.715226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46776 (local address 127.0.97.62:38821)
0619 01:10:40.715394 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.715398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.715499 (+ 101us) server_negotiation.cc:407] Connection header received
0619 01:10:40.715660 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.715663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.715726 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.715805 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:40.717246 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.718065 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.719714 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.720286 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.720440 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.723276 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:40.723281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:40.723285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:40.723325 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:40.725975 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:40.726743 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:40.726747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:40.726750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:40.726882 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:40.727200 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:40.727203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:40.727205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:40.727539 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:40.727729 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:40.728164 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 01:10:40.728350 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:10:40.732476 16461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:10:40.742302 16461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:10:40.781678 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.756138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35715 (local address 127.0.0.1:55334)
0619 01:10:40.757116 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:40.757159 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:40.757387 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:40.773037 (+ 15650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:40.773040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:40.773060 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:10:40.773771 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.773788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.775803 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.775807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.778018 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.778032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.779794 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.779803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.779998 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.780057 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 01:10:40.780219 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 01:10:40.780436 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":203,"thread_start_us":550,"threads_started":1}
I20260619 01:10:40.781834 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.756753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42087 (local address 127.0.0.1:43984)
0619 01:10:40.769047 (+ 12294us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:40.769083 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:40.769318 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:40.772965 (+ 3647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:40.772980 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:40.773018 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:10:40.773588 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.773608 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.775567 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.775571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.778537 (+ 2966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.778546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.780769 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.780775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.780992 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.781031 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 01:10:40.781141 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 01:10:40.781321 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12071,"mutex_wait_us":121,"thread_start_us":12722,"threads_started":1}
I20260619 01:10:40.782171 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.758336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.0.97.1:39611)
0619 01:10:40.770965 (+ 12629us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.770979 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.771856 (+ 877us) server_negotiation.cc:407] Connection header received
0619 01:10:40.772004 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.772014 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.772304 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.772668 (+ 364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:10:40.773628 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.775386 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.778706 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.780244 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.780359 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.780591 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:40.781355 (+ 764us) server_negotiation.cc:299] Negotiation successful
0619 01:10:40.781538 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12355,"thread_start_us":12167,"threads_started":1}
I20260619 01:10:40.782337 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.756696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.0.97.2:35715)
0619 01:10:40.772405 (+ 15709us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.772412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.772439 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:10:40.772528 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.772534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.772847 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.773074 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:10:40.773922 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.775635 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.778186 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.779530 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.779690 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.780009 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:40.780275 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 01:10:40.780446 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15496,"thread_start_us":15354,"threads_started":1}
I20260619 01:10:40.782502 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.757710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.0.1:36956)
0619 01:10:40.771508 (+ 13798us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:40.771533 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:40.771706 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:40.772695 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:40.772703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:40.772738 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:10:40.773430 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.773445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.775565 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.775570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.778504 (+ 2934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.778526 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.780943 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.780950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.781138 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.781169 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 01:10:40.781256 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 01:10:40.781414 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13636,"mutex_wait_us":260,"thread_start_us":13491,"threads_started":1}
I20260619 01:10:40.782908 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.757408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.0.97.3:42087)
0619 01:10:40.771981 (+ 14573us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.771990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.772020 (+ 30us) server_negotiation.cc:407] Connection header received
0619 01:10:40.772112 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.772118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.772302 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.772662 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:10:40.773807 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.775401 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.778712 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.780257 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.780372 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.780591 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:40.781205 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 01:10:40.782542 (+ 1337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14384,"thread_start_us":14219,"threads_started":1}
I20260619 01:10:40.786778 16581 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6b5d22fc3d4c74a2049b79e4edb086 (DEFAULT_TABLE table=test-table [id=38905bc0e2e54ec9bbd96bdd2988f4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:10:40.786891 16713 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6b5d22fc3d4c74a2049b79e4edb086 (DEFAULT_TABLE table=test-table [id=38905bc0e2e54ec9bbd96bdd2988f4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:10:40.786885 16844 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6b5d22fc3d4c74a2049b79e4edb086 (DEFAULT_TABLE table=test-table [id=38905bc0e2e54ec9bbd96bdd2988f4ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:10:40.788368 16844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6b5d22fc3d4c74a2049b79e4edb086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:10:40.788367 16713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6b5d22fc3d4c74a2049b79e4edb086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:10:40.788584 16581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6b5d22fc3d4c74a2049b79e4edb086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:10:40.802927 16925 tablet_bootstrap.cc:492] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Bootstrap starting.
I20260619 01:10:40.805446 16926 tablet_bootstrap.cc:492] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: Bootstrap starting.
I20260619 01:10:40.807248 16925 tablet_bootstrap.cc:654] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Neither blocks nor log segments found. Creating new log.
I20260619 01:10:40.808821 16925 log.cc:824] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Log is configured to *not* fsync() on all Append() calls
I20260619 01:10:40.810277 16927 tablet_bootstrap.cc:492] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: Bootstrap starting.
I20260619 01:10:40.813544 16926 tablet_bootstrap.cc:654] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: Neither blocks nor log segments found. Creating new log.
I20260619 01:10:40.814692 16926 log.cc:824] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: Log is configured to *not* fsync() on all Append() calls
I20260619 01:10:40.815409 16927 tablet_bootstrap.cc:654] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: Neither blocks nor log segments found. Creating new log.
I20260619 01:10:40.816807 16927 log.cc:824] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: Log is configured to *not* fsync() on all Append() calls
I20260619 01:10:40.817909 16926 tablet_bootstrap.cc:492] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: No bootstrap required, opened a new log
I20260619 01:10:40.818248 16926 ts_tablet_manager.cc:1397] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20260619 01:10:40.818297 16925 tablet_bootstrap.cc:492] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: No bootstrap required, opened a new log
I20260619 01:10:40.818631 16925 ts_tablet_manager.cc:1397] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.002s
I20260619 01:10:40.820974 16927 tablet_bootstrap.cc:492] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: No bootstrap required, opened a new log
I20260619 01:10:40.821245 16927 ts_tablet_manager.cc:1397] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260619 01:10:40.827255 16926 raft_consensus.cc:348] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:10:40.827740 16926 raft_consensus.cc:374] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:10:40.827878 16926 raft_consensus.cc:729] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79e1bb3913541228217340ebb478f76, State: Initialized, Role: FOLLOWER
I20260619 01:10:40.828711 16926 consensus_queue.cc:260] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:10:40.829947 16778 heartbeater.cc:503] Master 127.0.97.62:38821 was elected leader, sending a full tablet report...
I20260619 01:10:40.829833 16925 raft_consensus.cc:348] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:10:40.830291 16925 raft_consensus.cc:374] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:10:40.830385 16925 raft_consensus.cc:729] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a992b7f6c9843ecbcfcc095cff9d893, State: Initialized, Role: FOLLOWER
I20260619 01:10:40.830717 16926 ts_tablet_manager.cc:1428] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 01:10:40.831154 16925 consensus_queue.cc:260] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:10:40.832528 16927 raft_consensus.cc:348] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:10:40.833112 16927 raft_consensus.cc:374] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:10:40.833249 16927 raft_consensus.cc:729] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a972f4609f4ea0a0e835293b90f45c, State: Initialized, Role: FOLLOWER
I20260619 01:10:40.834110 16927 consensus_queue.cc:260] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:10:40.835723 16925 ts_tablet_manager.cc:1428] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260619 01:10:40.838171 16909 heartbeater.cc:503] Master 127.0.97.62:38821 was elected leader, sending a full tablet report...
I20260619 01:10:40.838914 16927 ts_tablet_manager.cc:1428] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260619 01:10:40.860482 16933 raft_consensus.cc:482] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:10:40.860800 16933 raft_consensus.cc:504] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:10:40.862607 16933 leader_election.cc:288] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715), 1a992b7f6c9843ecbcfcc095cff9d893 (127.0.97.1:39611)
I20260619 01:10:40.868544 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.863092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56621 (local address 127.0.97.2:35715)
0619 01:10:40.863292 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.863296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.863316 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:10:40.863558 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.863564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.863644 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.863746 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:10:40.864244 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.865057 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.866791 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.867594 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.867750 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.867845 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:40.868125 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 01:10:40.868270 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:10:40.868544 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.863021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35715 (local address 127.0.97.3:56621)
0619 01:10:40.863233 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:40.863255 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:40.863400 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:40.863905 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:40.863909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:40.863932 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:10:40.864086 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.864094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.865223 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.865229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.866630 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.866645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.867762 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.867767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.867972 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.867995 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 01:10:40.868099 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 01:10:40.868233 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 01:10:40.869685 16733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6b5d22fc3d4c74a2049b79e4edb086" candidate_uuid: "a0a972f4609f4ea0a0e835293b90f45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79e1bb3913541228217340ebb478f76" is_pre_election: true
I20260619 01:10:40.869954 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.863648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57475 (local address 127.0.97.1:39611)
0619 01:10:40.863910 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.863919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.864670 (+ 751us) server_negotiation.cc:407] Connection header received
0619 01:10:40.864858 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.864863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.864932 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.865030 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:10:40.865692 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.866433 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.868063 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.869013 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.869167 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.869273 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:40.869576 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 01:10:40.869713 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 01:10:40.870023 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.863465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.97.3:57475)
0619 01:10:40.864566 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:40.864580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 01:10:40.864718 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:10:40.865044 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:10:40.865049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:10:40.865073 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:10:40.865529 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.865538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.866577 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.866584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.867898 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:10:40.867909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.869218 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.869225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:10:40.869415 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.869439 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 01:10:40.869547 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 01:10:40.869676 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"thread_start_us":834,"threads_started":1}
I20260619 01:10:40.870249 16733 raft_consensus.cc:2454] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0a972f4609f4ea0a0e835293b90f45c in term 0.
I20260619 01:10:40.870998 16800 leader_election.cc:302] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a972f4609f4ea0a0e835293b90f45c, c79e1bb3913541228217340ebb478f76; no voters:
I20260619 01:10:40.871008 16601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6b5d22fc3d4c74a2049b79e4edb086" candidate_uuid: "a0a972f4609f4ea0a0e835293b90f45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" is_pre_election: true
I20260619 01:10:40.871428 16933 raft_consensus.cc:2781] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:10:40.871549 16933 raft_consensus.cc:482] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:10:40.871565 16601 raft_consensus.cc:2454] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0a972f4609f4ea0a0e835293b90f45c in term 0.
I20260619 01:10:40.871667 16933 raft_consensus.cc:3037] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:10:40.874161 16933 raft_consensus.cc:504] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:10:40.874866 16933 leader_election.cc:288] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [CANDIDATE]: Term 1 election: Requested vote from peers c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715), 1a992b7f6c9843ecbcfcc095cff9d893 (127.0.97.1:39611)
I20260619 01:10:40.875429 16601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6b5d22fc3d4c74a2049b79e4edb086" candidate_uuid: "a0a972f4609f4ea0a0e835293b90f45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a992b7f6c9843ecbcfcc095cff9d893"
I20260619 01:10:40.875516 16733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6b5d22fc3d4c74a2049b79e4edb086" candidate_uuid: "a0a972f4609f4ea0a0e835293b90f45c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79e1bb3913541228217340ebb478f76"
I20260619 01:10:40.875689 16601 raft_consensus.cc:3037] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:10:40.875737 16733 raft_consensus.cc:3037] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:10:40.878172 16601 raft_consensus.cc:2454] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a972f4609f4ea0a0e835293b90f45c in term 1.
I20260619 01:10:40.878175 16733 raft_consensus.cc:2454] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a972f4609f4ea0a0e835293b90f45c in term 1.
I20260619 01:10:40.878659 16800 leader_election.cc:302] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0a972f4609f4ea0a0e835293b90f45c, c79e1bb3913541228217340ebb478f76; no voters:
I20260619 01:10:40.878995 16933 raft_consensus.cc:2781] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:10:40.879462 16933 raft_consensus.cc:686] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 1 LEADER]: Becoming Leader. State: Replica: a0a972f4609f4ea0a0e835293b90f45c, State: Running, Role: LEADER
I20260619 01:10:40.880093 16933 consensus_queue.cc:237] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:10:40.885083 16461 catalog_manager.cc:5495] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c reported cstate change: term changed from 0 to 1, leader changed from <none> to a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a0a972f4609f4ea0a0e835293b90f45c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } health_report { overall_health: HEALTHY } } }
W20260619 01:10:40.922268 16910 tablet.cc:2369] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:10:40.926860 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.921453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.0.97.3:42087)
0619 01:10:40.921672 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.921677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.921699 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:10:40.921886 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.921890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.921982 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.922113 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:10:40.922614 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.923344 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.924659 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.925063 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.925162 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.925715 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:10:40.926255 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:10:40.926388 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:40.926472 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 01:10:40.926548 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 01:10:40.927290 16779 tablet.cc:2369] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:10:40.927801 16647 tablet.cc:2369] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:10:40.935875 16601 raft_consensus.cc:1261] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Refusing update from remote peer a0a972f4609f4ea0a0e835293b90f45c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:10:40.935926 16733 raft_consensus.cc:1261] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76 [term 1 FOLLOWER]: Refusing update from remote peer a0a972f4609f4ea0a0e835293b90f45c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:10:40.936940 16935 consensus_queue.cc:1035] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:10:40.937338 16933 consensus_queue.cc:1035] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:10:40.950999 16939 mvcc.cc:205] Tried to move back new op lower bound from 7298381582063099904 to 7298381581849206784. Current Snapshot: MvccSnapshot[applied={T|T < 7298381582063099904}]
I20260619 01:10:40.950996 16938 mvcc.cc:205] Tried to move back new op lower bound from 7298381582063099904 to 7298381581849206784. Current Snapshot: MvccSnapshot[applied={T|T < 7298381582063099904}]
I20260619 01:10:40.965098 16460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:10:40.965497 16460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:10:40.969532 16460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Sending DeleteTablet for 3 replicas of tablet 1c6b5d22fc3d4c74a2049b79e4edb086
I20260619 01:10:40.970927 16713 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6b5d22fc3d4c74a2049b79e4edb086 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:10:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55334
I20260619 01:10:40.970949 16844 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6b5d22fc3d4c74a2049b79e4edb086 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:10:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43984
I20260619 01:10:40.971135 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 16498
I20260619 01:10:40.972002 16581 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6b5d22fc3d4c74a2049b79e4edb086 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:10:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36956
I20260619 01:10:40.982414 16947 tablet_replica.cc:330] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: stopping tablet replica
I20260619 01:10:40.983137 16947 raft_consensus.cc:2229] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:10:40.984032 16947 raft_consensus.cc:2258] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:10:40.986783 16948 tablet_replica.cc:330] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: stopping tablet replica
I20260619 01:10:40.987319 16948 raft_consensus.cc:2229] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:10:40.987905 16948 raft_consensus.cc:2258] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:10:40.989430 16949 tablet_replica.cc:330] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: stopping tablet replica
I20260619 01:10:40.990036 16949 raft_consensus.cc:2229] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:10:40.990638 16949 raft_consensus.cc:2258] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:10:40.990943 16948 ts_tablet_manager.cc:1905] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:10:40.992552 16947 ts_tablet_manager.cc:1905] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:10:40.997411 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:40.972486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36972 (local address 127.0.97.1:39611)
0619 01:10:40.972744 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:40.972752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:40.972775 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:10:40.972861 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:40.972868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:40.973006 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:40.973126 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:40.975766 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.977431 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.981283 (+ 3852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:40.982305 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:40.982488 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:40.988205 (+ 5717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:40.988228 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:40.988256 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:40.988307 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:40.992852 (+ 4545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:40.993935 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:40.993945 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:40.993949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:40.994127 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:40.994599 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:40.994604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:40.994607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:40.995102 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:40.995411 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:40.996711 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0619 01:10:40.996972 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 01:10:40.998126 16641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:10:40.998662 16948 ts_tablet_manager.cc:1918] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:10:40.998903 16948 log.cc:1196] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/1c6b5d22fc3d4c74a2049b79e4edb086
I20260619 01:10:40.999572 16948 ts_tablet_manager.cc:1939] T 1c6b5d22fc3d4c74a2049b79e4edb086 P c79e1bb3913541228217340ebb478f76: Deleting consensus metadata
I20260619 01:10:41.002166 16448 catalog_manager.cc:4841] TS c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715): tablet 1c6b5d22fc3d4c74a2049b79e4edb086 (table test-table [id=38905bc0e2e54ec9bbd96bdd2988f4ea]) successfully deleted
I20260619 01:10:41.002485 16947 ts_tablet_manager.cc:1918] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:10:41.002779 16947 log.cc:1196] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/1c6b5d22fc3d4c74a2049b79e4edb086
I20260619 01:10:41.003388 16947 ts_tablet_manager.cc:1939] T 1c6b5d22fc3d4c74a2049b79e4edb086 P a0a972f4609f4ea0a0e835293b90f45c: Deleting consensus metadata
I20260619 01:10:41.005458 16445 catalog_manager.cc:4841] TS a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087): tablet 1c6b5d22fc3d4c74a2049b79e4edb086 (table test-table [id=38905bc0e2e54ec9bbd96bdd2988f4ea]) successfully deleted
I20260619 01:10:41.126961 16949 ts_tablet_manager.cc:1905] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:10:41.127784 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 16498
W20260619 01:10:41.152894 16448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39611 (error 108)
I20260619 01:10:41.153720 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 16650
I20260619 01:10:41.155097 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:41.154500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.0.1:36974)
0619 01:10:41.154778 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:41.154889 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260619 01:10:41.155396 16448 catalog_manager.cc:4584] TS 1a992b7f6c9843ecbcfcc095cff9d893 (127.0.97.1:39611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c6b5d22fc3d4c74a2049b79e4edb086: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39611: connect: Connection refused (error 111)
I20260619 01:10:41.170449 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:41.154548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.0.97.2:35715)
0619 01:10:41.154734 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:41.154741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:41.154946 (+ 205us) server_negotiation.cc:407] Connection header received
0619 01:10:41.155123 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:41.155130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:41.155229 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:41.155352 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:41.157180 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:41.158131 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:41.160043 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:41.160478 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:41.160586 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:41.164149 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:41.164166 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:41.164183 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:41.164248 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:41.167324 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:41.168327 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:41.168336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:41.168338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:41.168481 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:41.168909 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:41.168913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:41.168915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:41.169403 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:41.169641 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:41.170005 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 01:10:41.170179 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:10:41.171600 16773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:10:41.209914 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:41.208979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.0.1:36978)
0619 01:10:41.209578 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:41.209712 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436}
I20260619 01:10:41.211803 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:41.211285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.0.1:36994)
0619 01:10:41.211524 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:41.211631 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 01:10:41.266513 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 16650
I20260619 01:10:41.272711 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:41.272037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.0.1:36996)
0619 01:10:41.272330 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:41.272462 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 01:10:41.274082 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:41.273637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.0.1:37002)
0619 01:10:41.273830 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:41.273927 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 01:10:41.288805 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 16781
I20260619 01:10:41.306726 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:41.289785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.0.97.3:42087)
0619 01:10:41.289979 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 01:10:41.289984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:10:41.290175 (+ 191us) server_negotiation.cc:407] Connection header received
0619 01:10:41.290380 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:10:41.290386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:10:41.290481 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:10:41.290635 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:10:41.292307 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:41.293602 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:41.295063 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:10:41.295782 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:10:41.295938 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:10:41.298656 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:10:41.298674 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:10:41.298695 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:10:41.298751 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:10:41.303145 (+ 4394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:41.304259 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:41.304267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:41.304270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:41.304458 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:10:41.305001 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:10:41.305005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:10:41.305007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:10:41.305490 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:10:41.305789 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 01:10:41.306188 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 01:10:41.306377 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:10:41.307528 16904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:10:41.354769 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:41.354264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.0.1:37004)
0619 01:10:41.354499 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:41.354592 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 01:10:41.355990 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:41.355546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.0.1:37012)
0619 01:10:41.355787 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:41.355873 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 01:10:41.434667 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 16781
I20260619 01:10:41.455407 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 16428
I20260619 01:10:41.456650 16491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:10:41.572997 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:10:41.572369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39611 (local address 127.0.0.1:37026)
0619 01:10:41.572609 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 01:10:41.572751 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 01:10:41.573172 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 16428
I20260619 01:11:01.597136 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38821
--webserver_interface=127.0.97.62
--webserver_port=36171
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:01.941743 16955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:01.942070 16955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:01.942235 16955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:01.949617 16955 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 01:11:01.949733 16955 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 01:11:01.949772 16955 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 01:11:01.949882 16955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:01.949982 16955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:01.950098 16955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:01.950165 16955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:01.950234 16955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:01.960445 16955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:01.961886 16955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:01.963275 16955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:01.976881 16961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:01.976929 16960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:01.977113 16955 server_base.cc:1034] running on GCE node
W20260619 01:11:01.976902 16963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:01.978372 16955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:01.979861 16955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:01.981140 16955 hybrid_clock.cc:648] HybridClock initialized: now 1781831461981053 us; error 45 us; skew 500 ppm
I20260619 01:11:01.981613 16955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:01 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:01.984933 16955 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:11:01.986644 16955 webserver.cc:460] Webserver started at http://127.0.97.62:36171/ using document root <none> and password file <none>
I20260619 01:11:01.987406 16955 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:01.987540 16955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:01.993569 16955 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 01:11:01.996788 16970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:01.998397 16955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:01.998618 16955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "23df21a0905645d2a99a3b7e8c348ddb"
format_stamp: "Formatted at 2026-06-19 01:10:15 on dist-test-slave-5p8w"
server_key: "00ac694168a0d0e7afea4f52e4726776"
server_key_iv: "81b899a32bfb6b4bf52926a776567f4a"
server_key_version: "encryptionkey@0"
I20260619 01:11:01.999691 16955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:02.028478 16955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:02.032996 16955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:02.033376 16955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:02.053383 16955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38821
I20260619 01:11:02.053481 17021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38821 every 8 connection(s)
I20260619 01:11:02.055053 16955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:02.062506 17022 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:11:02.062903 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 16955
I20260619 01:11:02.063957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39611
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38821
--webserver_port=38783
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:11:02.068140 17022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Bootstrap starting.
I20260619 01:11:02.091629 17022 log.cc:824] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:02.109151 17022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:11:02.109802 17022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Bootstrap complete.
I20260619 01:11:02.119871 17022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:11:02.120807 17022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23df21a0905645d2a99a3b7e8c348ddb, State: Initialized, Role: FOLLOWER
I20260619 01:11:02.121680 17022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:11:02.121989 17022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:02.122118 17022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:02.122313 17022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 2 FOLLOWER]: Advancing to term 3
I20260619 01:11:02.126431 17022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:11:02.126971 17022 leader_election.cc:302] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23df21a0905645d2a99a3b7e8c348ddb; no voters:
I20260619 01:11:02.127692 17022 leader_election.cc:288] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 01:11:02.127849 17026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 3 FOLLOWER]: Leader election won for term 3
I20260619 01:11:02.129107 17026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [term 3 LEADER]: Becoming Leader. State: Replica: 23df21a0905645d2a99a3b7e8c348ddb, State: Running, Role: LEADER
I20260619 01:11:02.129702 17026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } }
I20260619 01:11:02.130036 17022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:02.132477 17027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "23df21a0905645d2a99a3b7e8c348ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } } }
I20260619 01:11:02.132931 17027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:02.133375 17028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23df21a0905645d2a99a3b7e8c348ddb. Latest consensus state: current_term: 3 leader_uuid: "23df21a0905645d2a99a3b7e8c348ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23df21a0905645d2a99a3b7e8c348ddb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38821 } } }
I20260619 01:11:02.133829 17028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:02.141670 17031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:02.152136 17031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:02.153765 17031 catalog_manager.cc:1259] Loaded cluster ID: c5b6de766ad44ac7b2518e413ec8af24
I20260619 01:11:02.153898 17031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:02.158142 17031 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:02.160753 17031 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Loaded TSK: 0
I20260619 01:11:02.160944 17031 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23df21a0905645d2a99a3b7e8c348ddb: Loaded TSK: 1
F20260619 01:11:02.162199 17031 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcb84dea9ad google::LogMessage::Fail() at ??:0
@ 0x7fcb84ded88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcb84dea4de google::LogMessage::Flush() at ??:0
@ 0x7fcb84deba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcb88b4e7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcb88b3d73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fcb9a8191c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcb9a818783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcb9a8a42a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fcb9a8a40c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fcb9a8ee5e9 std::function<>::operator()() at ??:0
@ 0x7fcb9a81ccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcb9a81ac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcb9a8a2610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcb9a8a2439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcb87a36345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcb87a3991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fcb87a39799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcb87a14eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcb8acee6db start_thread at ??:0
@ 0x7fcb83c8461f clone at ??:0
W20260619 01:11:02.479337 17024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:02.479722 17024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:02.479915 17024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:02.488858 17024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:02.488996 17024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:02.489228 17024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:11:02.499720 17024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38783
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:02.501168 17024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:02.502632 17024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:02.516793 17069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:02.516877 17072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:02.516832 17070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:02.517910 17024 server_base.cc:1034] running on GCE node
I20260619 01:11:02.518641 17024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:02.519730 17024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:02.520977 17024 hybrid_clock.cc:648] HybridClock initialized: now 1781831462520906 us; error 68 us; skew 500 ppm
I20260619 01:11:02.521389 17024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:02 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:02.524892 17024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:11:02.526605 17024 webserver.cc:460] Webserver started at http://127.0.97.1:38783/ using document root <none> and password file <none>
I20260619 01:11:02.527385 17024 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:02.527549 17024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:02.533649 17024 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:11:02.536959 17079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:02.538558 17024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:02.538784 17024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "1a992b7f6c9843ecbcfcc095cff9d893"
format_stamp: "Formatted at 2026-06-19 01:10:16 on dist-test-slave-5p8w"
server_key: "2926ba4e97fba9e014098dc4866f310e"
server_key_iv: "c4f81fd8745c188387eedcc6cd6073cc"
server_key_version: "encryptionkey@0"
I20260619 01:11:02.539922 17024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:02.554422 17024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:02.559427 17024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:02.559758 17024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:02.560848 17024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:02.563057 17086 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 01:11:02.569309 17024 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 01:11:02.569507 17024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260619 01:11:02.569674 17024 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 01:11:02.573523 17024 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 01:11:02.573647 17024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20260619 01:11:02.574035 17086 tablet_bootstrap.cc:492] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Bootstrap starting.
I20260619 01:11:02.584265 17086 log.cc:824] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:02.601500 17086 tablet_bootstrap.cc:492] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:11:02.602236 17086 tablet_bootstrap.cc:492] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Bootstrap complete.
I20260619 01:11:02.602775 17086 ts_tablet_manager.cc:1397] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.006s
I20260619 01:11:02.612753 17086 raft_consensus.cc:348] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:11:02.614542 17086 raft_consensus.cc:729] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a992b7f6c9843ecbcfcc095cff9d893, State: Initialized, Role: FOLLOWER
I20260619 01:11:02.615375 17086 consensus_queue.cc:260] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:11:02.617141 17086 ts_tablet_manager.cc:1428] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260619 01:11:02.621881 17024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39611
I20260619 01:11:02.621948 17194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39611 every 8 connection(s)
I20260619 01:11:02.623147 17024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:11:02.623998 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 17024
I20260619 01:11:02.624823 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35715
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38821
--webserver_port=34625
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:11:02.628520 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:02.626102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:34963)
0619 01:11:02.627073 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:02.627316 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":378,"threads_started":1}
W20260619 01:11:02.629365 17195 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38821 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38821: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
I20260619 01:11:02.630411 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:02.629949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:55401)
0619 01:11:02.630155 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:02.630253 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 01:11:02.631436 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:02.630981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:36059)
0619 01:11:02.631165 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:02.631267 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 01:11:02.631735 17195 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:11:02.973261 17198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:02.973654 17198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:02.973824 17198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:02.981725 17198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:02.981875 17198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:02.982069 17198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:11:02.992738 17198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34625
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:02.994302 17198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:02.995726 17198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:03.009922 17204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:03.009956 17203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:03.010010 17206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:03.010962 17198 server_base.cc:1034] running on GCE node
I20260619 01:11:03.011667 17198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:03.012655 17198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:03.013895 17198 hybrid_clock.cc:648] HybridClock initialized: now 1781831463013846 us; error 45 us; skew 500 ppm
I20260619 01:11:03.014271 17198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:03 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:03.017602 17198 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:11:03.019322 17198 webserver.cc:460] Webserver started at http://127.0.97.2:34625/ using document root <none> and password file <none>
I20260619 01:11:03.020120 17198 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:03.020273 17198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:03.026199 17198 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 01:11:03.029098 17213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:03.030644 17198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:11:03.030861 17198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "c79e1bb3913541228217340ebb478f76"
format_stamp: "Formatted at 2026-06-19 01:10:16 on dist-test-slave-5p8w"
server_key: "189b9452a1385f7cc7ea0e9367ecece7"
server_key_iv: "a8db51466d4918e3d5883e9d5a302f89"
server_key_version: "encryptionkey@0"
I20260619 01:11:03.031961 17198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:03.055070 17198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:03.059479 17198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:03.059787 17198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:03.060892 17198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:03.062446 17198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:03.062582 17198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:03.062726 17198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:03.062819 17198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:03.097257 17198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35715
I20260619 01:11:03.097376 17325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35715 every 8 connection(s)
I20260619 01:11:03.098805 17198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:11:03.101997 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 17198
I20260619 01:11:03.102664 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42087
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38821
--webserver_port=40643
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:11:03.104032 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:03.101848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:35287)
0619 01:11:03.102758 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:03.103041 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":392,"threads_started":1}
W20260619 01:11:03.104724 17326 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38821 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38821: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
I20260619 01:11:03.105751 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:03.105402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:33465)
0619 01:11:03.105564 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:03.105634 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 01:11:03.106714 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:03.106407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:38421)
0619 01:11:03.106533 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:03.106613 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 01:11:03.107012 17326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:11:03.453595 17329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:03.453903 17329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:03.454071 17329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:03.461582 17329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:03.461709 17329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:03.461911 17329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:11:03.472010 17329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40643
--tserver_master_addrs=127.0.97.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:03.473452 17329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:03.474781 17329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:03.488512 17334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:03.488597 17335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:03.488559 17337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:03.489812 17329 server_base.cc:1034] running on GCE node
I20260619 01:11:03.490536 17329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:03.491519 17329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:03.492733 17329 hybrid_clock.cc:648] HybridClock initialized: now 1781831463492678 us; error 47 us; skew 500 ppm
I20260619 01:11:03.493034 17329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:03 dist-test-slave-5p8w krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:03.496356 17329 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:11:03.498030 17329 webserver.cc:460] Webserver started at http://127.0.97.3:40643/ using document root <none> and password file <none>
I20260619 01:11:03.498797 17329 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:03.498934 17329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:03.504773 17329 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 01:11:03.508024 17344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:03.509752 17329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 01:11:03.509977 17329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "a0a972f4609f4ea0a0e835293b90f45c"
format_stamp: "Formatted at 2026-06-19 01:10:17 on dist-test-slave-5p8w"
server_key: "7200ac25c846d1936b691495d08d5dd9"
server_key_iv: "4a6ff560983719a4af0230ed5dd4b200"
server_key_version: "encryptionkey@0"
I20260619 01:11:03.511122 17329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:03.531080 17329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:03.535343 17329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:03.535651 17329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:03.536767 17329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:03.538286 17329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:03.538412 17329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:03.538553 17329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:03.538641 17329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:03.573362 17329 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42087
I20260619 01:11:03.573460 17456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42087 every 8 connection(s)
I20260619 01:11:03.575091 17329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:11:03.579658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 17329
I20260619 01:11:03.580803 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:03.578390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:46507)
0619 01:11:03.579254 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:03.579542 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":322,"threads_started":1}
W20260619 01:11:03.581645 17457 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38821 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38821: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
I20260619 01:11:03.582527 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:03.582198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:42059)
0619 01:11:03.582345 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:03.582417 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 01:11:03.583405 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:03.583100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:35425)
0619 01:11:03.583221 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:03.583296 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 01:11:03.583710 17457 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 01:11:03.633188 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:03.632373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:55805)
0619 01:11:03.632861 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:03.632951 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":208,"threads_started":1}
I20260619 01:11:04.096114 17462 raft_consensus.cc:482] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:11:04.096451 17462 raft_consensus.cc:504] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:11:04.098290 17462 leader_election.cc:288] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715), a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087)
I20260619 01:11:04.108577 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:04.107517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:45247)
0619 01:11:04.108062 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:04.108214 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":261,"threads_started":1}
Jun 19 01:11:04 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Jun 19 01:11:04 dist-test-slave-5p8w krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:11:04.131000 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:04.098818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42087 (local address 127.0.97.1:60209)
0619 01:11:04.106789 (+ 7971us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:04.106818 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:04.107015 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:04.110861 (+ 3846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:04.110873 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:04.111762 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:04.113749 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:04.113766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:04.115563 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:04.115568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:04.116634 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:04.116648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:04.117922 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:04.117928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:04.118165 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:04.118751 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:11:04.118769 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:04.121696 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:04.126265 (+ 4569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:04.126277 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:04.126297 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:04.127234 (+ 937us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:04.127864 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:04.127870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:04.127874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:04.128241 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:04.129409 (+ 1168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:04.129419 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:04.129927 (+ 508us) client_negotiation.cc:769] Sending connection context
0619 01:11:04.130255 (+ 328us) client_negotiation.cc:240] Negotiation successful
0619 01:11:04.130676 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7750,"mutex_wait_us":52,"spinlock_wait_cycles":66688,"thread_start_us":7485,"threads_started":1}
I20260619 01:11:04.131175 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:04.099106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60209 (local address 127.0.97.3:42087)
0619 01:11:04.107664 (+ 8558us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:04.107687 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:04.107744 (+ 57us) server_negotiation.cc:407] Connection header received
0619 01:11:04.109792 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:04.109805 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:04.110281 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:04.110670 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:04.113963 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:04.115349 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:04.116790 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:04.117713 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:04.117909 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:04.121979 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:04.121991 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:04.122009 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:04.122070 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:04.126033 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:04.127407 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:04.127416 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:04.127426 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:04.127673 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:04.128388 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:04.128392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:04.128396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:04.129221 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:04.129448 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:04.130489 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0619 01:11:04.130843 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8341,"thread_start_us":8208,"threads_started":1}
I20260619 01:11:04.132375 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:04.098643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35715 (local address 127.0.97.1:50643)
0619 01:11:04.099411 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:04.099449 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:04.100003 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:04.110862 (+ 10859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:04.110873 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:04.111765 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:04.112620 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:04.112669 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:04.114801 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:04.114806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:04.116267 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:04.116280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:04.117382 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:04.117389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:04.117710 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:04.118587 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:11:04.118618 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:04.121404 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:04.126265 (+ 4861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:04.126277 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:04.126297 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:04.127234 (+ 937us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:04.127864 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:04.127870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:04.127874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:04.128232 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:04.131295 (+ 3063us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:04.131302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:04.131503 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 01:11:04.131710 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 01:11:04.131994 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":56,"spinlock_wait_cycles":4608}
I20260619 01:11:04.132735 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:04.099251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50643 (local address 127.0.97.2:35715)
0619 01:11:04.108020 (+ 8769us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:04.108032 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:04.108070 (+ 38us) server_negotiation.cc:407] Connection header received
0619 01:11:04.109966 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:04.109974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:04.110327 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:04.110675 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:04.112920 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:04.114593 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:04.116438 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:04.117126 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:04.117361 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:04.121772 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:04.121786 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:04.121810 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:04.121865 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:04.126033 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:04.127406 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:04.127416 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:04.127426 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:04.127687 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:04.128376 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:04.128381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:04.128384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:04.129213 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:04.129448 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:04.131962 (+ 2514us) server_negotiation.cc:299] Negotiation successful
0619 01:11:04.132333 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7699,"thread_start_us":7558,"threads_started":1}
W20260619 01:11:04.134895 17080 leader_election.cc:341] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087): Not found: Tablet not found: 1c6b5d22fc3d4c74a2049b79e4edb086
W20260619 01:11:04.135643 17083 leader_election.cc:341] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715): Not found: Tablet not found: 1c6b5d22fc3d4c74a2049b79e4edb086
I20260619 01:11:04.135768 17083 leader_election.cc:302] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a992b7f6c9843ecbcfcc095cff9d893; no voters: a0a972f4609f4ea0a0e835293b90f45c, c79e1bb3913541228217340ebb478f76
I20260619 01:11:04.136159 17462 raft_consensus.cc:2726] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 01:11:04.585104 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:04.584236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:51155)
0619 01:11:04.584711 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:04.584844 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":216,"threads_started":1}
I20260619 01:11:04.634918 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:04.634225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:56025)
0619 01:11:04.634658 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:04.634753 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":214,"threads_started":1}
I20260619 01:11:05.110338 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:05.109617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:33937)
0619 01:11:05.110069 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:05.110160 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":235,"threads_started":1}
I20260619 01:11:05.586863 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:05.586109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:59517)
0619 01:11:05.586603 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:05.586697 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260619 01:11:05.636677 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:05.635905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:37923)
0619 01:11:05.636401 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:05.636484 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":277,"threads_started":1}
I20260619 01:11:06.112285 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:06.111422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:46119)
0619 01:11:06.112000 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:06.112087 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":331,"threads_started":1}
I20260619 01:11:06.248509 17473 raft_consensus.cc:482] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:11:06.248698 17473 raft_consensus.cc:504] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:11:06.249348 17473 leader_election.cc:288] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715), a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087)
W20260619 01:11:06.250371 17083 leader_election.cc:341] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715): Not found: Tablet not found: 1c6b5d22fc3d4c74a2049b79e4edb086
W20260619 01:11:06.250542 17080 leader_election.cc:341] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087): Not found: Tablet not found: 1c6b5d22fc3d4c74a2049b79e4edb086
I20260619 01:11:06.250780 17080 leader_election.cc:302] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a992b7f6c9843ecbcfcc095cff9d893; no voters: a0a972f4609f4ea0a0e835293b90f45c, c79e1bb3913541228217340ebb478f76
I20260619 01:11:06.251037 17473 raft_consensus.cc:2726] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 01:11:06.588862 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:06.587977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:56711)
0619 01:11:06.588593 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:06.588690 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":365,"threads_started":1}
I20260619 01:11:06.638516 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:06.637712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:46965)
0619 01:11:06.638214 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:06.638325 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":220,"threads_started":1}
I20260619 01:11:07.114285 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:07.113396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:54119)
0619 01:11:07.113919 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:07.114023 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":273,"threads_started":1}
I20260619 01:11:07.590708 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:07.589979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:47227)
0619 01:11:07.590435 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:07.590520 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260619 01:11:07.640456 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:07.639590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:56821)
0619 01:11:07.640142 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:07.640246 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":253,"threads_started":1}
I20260619 01:11:08.116401 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:08.115502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:46657)
0619 01:11:08.116040 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:08.116125 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":300,"threads_started":1}
I20260619 01:11:08.592446 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:08.591752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:47865)
0619 01:11:08.592192 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:08.592285 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":229,"threads_started":1}
I20260619 01:11:08.642397 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:08.641628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:51957)
0619 01:11:08.642108 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:08.642200 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260619 01:11:08.697011 17482 raft_consensus.cc:482] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:11:08.697217 17482 raft_consensus.cc:504] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:11:08.697881 17482 leader_election.cc:288] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715), a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087)
W20260619 01:11:08.698913 17080 leader_election.cc:341] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087): Not found: Tablet not found: 1c6b5d22fc3d4c74a2049b79e4edb086
W20260619 01:11:08.699106 17083 leader_election.cc:341] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715): Not found: Tablet not found: 1c6b5d22fc3d4c74a2049b79e4edb086
I20260619 01:11:08.699224 17083 leader_election.cc:302] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a992b7f6c9843ecbcfcc095cff9d893; no voters: a0a972f4609f4ea0a0e835293b90f45c, c79e1bb3913541228217340ebb478f76
I20260619 01:11:08.699457 17482 raft_consensus.cc:2726] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 01:11:09.118283 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:09.117466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:60183)
0619 01:11:09.117956 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:09.118051 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260619 01:11:09.594341 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:09.593543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:49951)
0619 01:11:09.594048 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:09.594137 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":266,"threads_started":1}
I20260619 01:11:09.644505 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:09.643613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:59631)
0619 01:11:09.644200 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:09.644307 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":268,"threads_started":1}
I20260619 01:11:10.120249 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:10.119417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:53595)
0619 01:11:10.119944 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:10.120040 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":228,"threads_started":1}
I20260619 01:11:10.596164 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:10.595341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:49523)
0619 01:11:10.595883 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:10.595973 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":252,"threads_started":1}
I20260619 01:11:10.646435 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:10.645585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:45421)
0619 01:11:10.646157 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:10.646248 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":314,"threads_started":1}
I20260619 01:11:11.122117 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:11.121260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:41455)
0619 01:11:11.121850 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:11.121947 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":301,"threads_started":1}
I20260619 01:11:11.598047 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:11.597278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:58559)
0619 01:11:11.597764 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:11.597850 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":211,"threads_started":1}
I20260619 01:11:11.648238 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:11.647455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:40303)
0619 01:11:11.647966 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:11.648060 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":248,"threads_started":1}
I20260619 01:11:12.047250 17492 raft_consensus.cc:482] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:11:12.047612 17492 raft_consensus.cc:504] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e1bb3913541228217340ebb478f76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35715 } } peers { permanent_uuid: "1a992b7f6c9843ecbcfcc095cff9d893" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39611 } } peers { permanent_uuid: "a0a972f4609f4ea0a0e835293b90f45c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42087 } }
I20260619 01:11:12.048265 17492 leader_election.cc:288] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715), a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087)
W20260619 01:11:12.049306 17080 leader_election.cc:341] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0a972f4609f4ea0a0e835293b90f45c (127.0.97.3:42087): Not found: Tablet not found: 1c6b5d22fc3d4c74a2049b79e4edb086
W20260619 01:11:12.049492 17083 leader_election.cc:341] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c79e1bb3913541228217340ebb478f76 (127.0.97.2:35715): Not found: Tablet not found: 1c6b5d22fc3d4c74a2049b79e4edb086
I20260619 01:11:12.049652 17083 leader_election.cc:302] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a992b7f6c9843ecbcfcc095cff9d893; no voters: a0a972f4609f4ea0a0e835293b90f45c, c79e1bb3913541228217340ebb478f76
I20260619 01:11:12.049914 17492 raft_consensus.cc:2726] T 1c6b5d22fc3d4c74a2049b79e4edb086 P 1a992b7f6c9843ecbcfcc095cff9d893 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 01:11:12.123929 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:12.123117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:53919)
0619 01:11:12.123653 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:12.123753 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":283,"threads_started":1}
I20260619 01:11:12.599951 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:12.599114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:51625)
0619 01:11:12.599648 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:12.599768 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":271,"threads_started":1}
I20260619 01:11:12.650238 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:12.649476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:41331)
0619 01:11:12.649969 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:12.650068 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":254,"threads_started":1}
I20260619 01:11:13.125790 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:13.124972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:39073)
0619 01:11:13.125503 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:13.125596 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":251,"threads_started":1}
I20260619 01:11:13.601895 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:13.601096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:58997)
0619 01:11:13.601605 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:13.601693 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":283,"threads_started":1}
I20260619 01:11:13.652097 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:13.651288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:55173)
0619 01:11:13.651820 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:13.651907 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":278,"threads_started":1}
I20260619 01:11:14.127604 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:14.126803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:56181)
0619 01:11:14.127328 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:14.127425 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":256,"threads_started":1}
I20260619 01:11:14.603741 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:14.602901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:54647)
0619 01:11:14.603490 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:14.603581 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":324,"threads_started":1}
I20260619 01:11:14.653927 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:14.653135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:41335)
0619 01:11:14.653656 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:14.653746 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":302,"threads_started":1}
I20260619 01:11:15.129652 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:15.128610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:54941)
0619 01:11:15.129186 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:15.129372 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":266,"threads_started":1}
I20260619 01:11:15.605540 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:15.604754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:54323)
0619 01:11:15.605259 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:15.605368 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":277,"threads_started":1}
I20260619 01:11:15.655925 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:15.654990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:47279)
0619 01:11:15.655632 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:15.655725 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":353,"threads_started":1}
I20260619 01:11:16.131316 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:16.130619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:39909)
0619 01:11:16.131056 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:16.131133 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":232,"threads_started":1}
I20260619 01:11:16.607364 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:16.606593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:43877)
0619 01:11:16.607074 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:16.607173 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":242,"threads_started":1}
I20260619 01:11:16.657998 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:16.657162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:54241)
0619 01:11:16.657704 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:16.657813 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":299,"threads_started":1}
I20260619 01:11:17.137952 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:17.136859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:55237)
0619 01:11:17.137486 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:17.137623 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":339,"threads_started":1}
I20260619 01:11:17.609243 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:17.608397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:37037)
0619 01:11:17.608917 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:17.609015 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":283,"threads_started":1}
I20260619 01:11:17.660018 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:17.659233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.1:42311)
0619 01:11:17.659725 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:17.659820 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":244,"threads_started":1}
I20260619 01:11:18.140077 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:18.139242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.2:53487)
0619 01:11:18.139830 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:18.139916 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":361,"threads_started":1}
I20260619 01:11:18.592573 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 17024
I20260619 01:11:18.611341 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:18.610589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38821 (local address 127.0.97.3:43145)
0619 01:11:18.611079 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:18.611169 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
I20260619 01:11:18.617024 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:18.593922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44894 (local address 127.0.97.1:39611)
0619 01:11:18.594489 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:18.594496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:18.594520 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:11:18.594587 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:18.594594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:18.594823 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:18.595009 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:18.596460 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:18.597528 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:18.598872 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:18.599574 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:18.599729 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:18.610411 (+ 10682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:18.610438 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:18.610462 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:18.610512 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:18.613891 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:18.614748 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:18.614756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:18.614759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:18.614909 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:18.615321 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:18.615326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:18.615328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:18.615739 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:18.616071 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:18.616417 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 01:11:18.616693 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":268,"threads_started":1}
I20260619 01:11:18.618486 17190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:18.715835 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 17024
I20260619 01:11:18.734172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 17198
I20260619 01:11:18.748893 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:18.735117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.0.97.2:35715)
0619 01:11:18.735681 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:18.735685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:18.735701 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:11:18.735757 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:18.735760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:18.735825 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:18.735956 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:18.737539 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:18.738355 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:18.739678 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:18.740119 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:18.740270 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:18.743155 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:18.743161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:18.743164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:18.743202 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:18.745990 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:18.746748 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:18.746754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:18.746757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:18.746910 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:18.747327 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:18.747332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:18.747338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:18.747641 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:18.747947 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:18.748256 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 01:11:18.748505 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":295,"threads_started":1}
I20260619 01:11:18.749859 17321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:18.836748 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 17198
I20260619 01:11:18.855310 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 17329
I20260619 01:11:18.869100 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:18.856224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.0.97.3:42087)
0619 01:11:18.856843 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:18.856848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:18.856867 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:11:18.856923 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:18.856927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:18.856996 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:18.857098 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:18.858559 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:18.859358 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:18.860615 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:18.861022 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:18.861119 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:18.863892 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:18.863896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:18.863899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:18.863932 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:18.866743 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:18.867440 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:18.867444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:18.867446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:18.867599 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:18.867912 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:18.867915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:18.867917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:18.868171 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:18.868379 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:18.868630 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 01:11:18.868796 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":361,"threads_started":1}
I20260619 01:11:18.869840 17452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:18.953173 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 17329
2026-06-19T01:11:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65108 ms)
[----------] 30 tests from SecurityITest (307625 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:19 dist-test-slave-5p8w krb5kdc[17522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:19 dist-test-slave-5p8w krb5kdc[17522](info): set up 2 sockets
Jun 19 01:11:19 dist-test-slave-5p8w krb5kdc[17522](info): commencing operation
krb5kdc: starting...
W20260619 01:11:20.065569 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:20 dist-test-slave-5p8w krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:11:20.321621 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35567
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:20.678584 17535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:20.678936 17535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:20.679039 17535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:20.686738 17535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:20.686887 17535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:20.686941 17535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:20.687008 17535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:20.687067 17535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:20.697671 17535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:20.699074 17535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:20.700496 17535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:20.714679 17540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:20.714594 17543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:20.714608 17541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:20.715406 17535 server_base.cc:1034] running on GCE node
I20260619 01:11:20.716418 17535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:20.718134 17535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:20.719383 17535 hybrid_clock.cc:648] HybridClock initialized: now 1781831480719336 us; error 40 us; skew 500 ppm
I20260619 01:11:20.719812 17535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:20 dist-test-slave-5p8w krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:20.723217 17535 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:11:20.724983 17535 webserver.cc:460] Webserver started at http://127.0.97.62:38473/ using document root <none> and password file <none>
I20260619 01:11:20.725831 17535 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:20.725972 17535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:20.726334 17535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:20.729707 17535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "f1e146337da54293b3d19f0067686e98"
format_stamp: "Formatted at 2026-06-19 01:11:20 on dist-test-slave-5p8w"
server_key: "df1ec0872867ccb6eee0a02a448888ea"
server_key_iv: "d55d9cdb8f2cd79674a5f65b73f2802e"
server_key_version: "encryptionkey@0"
I20260619 01:11:20.730576 17535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "f1e146337da54293b3d19f0067686e98"
format_stamp: "Formatted at 2026-06-19 01:11:20 on dist-test-slave-5p8w"
server_key: "df1ec0872867ccb6eee0a02a448888ea"
server_key_iv: "d55d9cdb8f2cd79674a5f65b73f2802e"
server_key_version: "encryptionkey@0"
I20260619 01:11:20.736544 17535 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 01:11:20.740630 17550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:20.742349 17535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:20.742576 17535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "f1e146337da54293b3d19f0067686e98"
format_stamp: "Formatted at 2026-06-19 01:11:20 on dist-test-slave-5p8w"
server_key: "df1ec0872867ccb6eee0a02a448888ea"
server_key_iv: "d55d9cdb8f2cd79674a5f65b73f2802e"
server_key_version: "encryptionkey@0"
I20260619 01:11:20.742831 17535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:20.762889 17535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:20.767351 17535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:20.767678 17535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:20.787968 17535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35567
I20260619 01:11:20.788009 17601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35567 every 8 connection(s)
I20260619 01:11:20.789598 17535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:20.793887 17602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:20.797678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 17535
I20260619 01:11:20.797935 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:20.798763 388 external_mini_cluster.cc:1428] Setting key f534eaad024de69cc4ca8a006ea2a2c0
I20260619 01:11:20.807607 17602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98: Bootstrap starting.
I20260619 01:11:20.811076 17602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:20.812227 17602 log.cc:824] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98: Log is configured to *not* fsync() on all Append() calls
Jun 19 01:11:20 dist-test-slave-5p8w krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:20.815896 17602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98: No bootstrap required, opened a new log
I20260619 01:11:20.823331 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:20.801163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.0.97.62:35567)
0619 01:11:20.802046 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:20.802060 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:20.802107 (+ 47us) server_negotiation.cc:407] Connection header received
0619 01:11:20.803923 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:20.803951 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:20.804443 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:20.804903 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:20.806495 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:20.807917 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:20.809338 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:20.810066 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:20.810272 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:20.815185 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:20.815193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:20.815209 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:20.815252 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:20.818650 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:20.819529 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:20.819537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:20.819544 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:20.819734 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:20.820157 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:20.820164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:20.820167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:20.820768 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:20.821117 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:20.821636 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 01:11:20.822158 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":378,"threads_started":1}
I20260619 01:11:20.824860 17602 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e146337da54293b3d19f0067686e98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260619 01:11:20.825281 17602 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:20.825439 17602 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e146337da54293b3d19f0067686e98, State: Initialized, Role: FOLLOWER
I20260619 01:11:20.826072 17602 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e146337da54293b3d19f0067686e98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260619 01:11:20.826455 17602 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:20.826620 17602 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:20.826814 17602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:20.829154 17602 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e146337da54293b3d19f0067686e98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260619 01:11:20.829797 17602 leader_election.cc:302] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1e146337da54293b3d19f0067686e98; no voters:
I20260619 01:11:20.830401 17602 leader_election.cc:288] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:20.830561 17607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:20.831378 17607 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [term 1 LEADER]: Becoming Leader. State: Replica: f1e146337da54293b3d19f0067686e98, State: Running, Role: LEADER
I20260619 01:11:20.832059 17607 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e146337da54293b3d19f0067686e98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } }
I20260619 01:11:20.832804 17602 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:20.835632 17609 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1e146337da54293b3d19f0067686e98. Latest consensus state: current_term: 1 leader_uuid: "f1e146337da54293b3d19f0067686e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e146337da54293b3d19f0067686e98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } } }
I20260619 01:11:20.835268 17608 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1e146337da54293b3d19f0067686e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e146337da54293b3d19f0067686e98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35567 } } }
I20260619 01:11:20.836292 17609 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:20.836439 17608 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:20.838727 17615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:20.846249 17615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:20.856624 17615 catalog_manager.cc:1347] Generated new cluster ID: 604b9c9e3cff466199c7252f4a3636f8
I20260619 01:11:20.856768 17615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:20.867179 17615 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:20.868621 17615 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:20.876199 17615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98: Generated new TSK 0
I20260619 01:11:20.877130 17615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:11:20.940585 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:21.346800 17628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:21.347179 17628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:21.347393 17628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:21.355264 17628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:21.355412 17628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:21.355597 17628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:11:21.367286 17628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:21.368768 17628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:21.370349 17628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:21.385329 17636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:21.385407 17633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:21.385370 17634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:21.386685 17628 server_base.cc:1034] running on GCE node
I20260619 01:11:21.387370 17628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:21.388465 17628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:21.389773 17628 hybrid_clock.cc:648] HybridClock initialized: now 1781831481389712 us; error 50 us; skew 500 ppm
I20260619 01:11:21.390250 17628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:21 dist-test-slave-5p8w krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:21.394349 17628 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:11:21.396066 17628 webserver.cc:460] Webserver started at http://127.0.97.1:46431/ using document root <none> and password file <none>
I20260619 01:11:21.396895 17628 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:21.397061 17628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:21.397491 17628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:21.400759 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede"
format_stamp: "Formatted at 2026-06-19 01:11:21 on dist-test-slave-5p8w"
server_key: "b57b7e01c5f15d7d8b4207f727a6a094"
server_key_iv: "7b4bb21b6a1ac98b38a23609f903fbbd"
server_key_version: "encryptionkey@0"
I20260619 01:11:21.401759 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede"
format_stamp: "Formatted at 2026-06-19 01:11:21 on dist-test-slave-5p8w"
server_key: "b57b7e01c5f15d7d8b4207f727a6a094"
server_key_iv: "7b4bb21b6a1ac98b38a23609f903fbbd"
server_key_version: "encryptionkey@0"
I20260619 01:11:21.407649 17628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:11:21.411729 17643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:21.413525 17628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:21.413791 17628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede"
format_stamp: "Formatted at 2026-06-19 01:11:21 on dist-test-slave-5p8w"
server_key: "b57b7e01c5f15d7d8b4207f727a6a094"
server_key_iv: "7b4bb21b6a1ac98b38a23609f903fbbd"
server_key_version: "encryptionkey@0"
I20260619 01:11:21.414073 17628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:21.452989 17628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:21.458086 17628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:21.458468 17628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:21.459594 17628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:21.461253 17628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:21.461406 17628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 01:11:21.461524 17628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:21.461572 17628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:21.495908 17628 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38779
I20260619 01:11:21.495955 17755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38779 every 8 connection(s)
I20260619 01:11:21.497398 17628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:11:21.502486 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 17628
I20260619 01:11:21.502772 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:11:21.503422 388 external_mini_cluster.cc:1428] Setting key 9f51542befdb7757a1682ddd0d8c8abe
Jun 19 01:11:21 dist-test-slave-5p8w krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:21.522295 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:21.500266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52811 (local address 127.0.97.62:35567)
0619 01:11:21.500773 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:21.500780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:21.501932 (+ 1152us) server_negotiation.cc:407] Connection header received
0619 01:11:21.504064 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:21.504070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:21.504176 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:21.504309 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:21.506100 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:21.506856 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:21.508573 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:21.509036 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:21.509190 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:21.513137 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:21.513143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:21.513146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:21.513185 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:21.517645 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:21.518843 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:21.518854 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:21.518861 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:21.519083 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:21.519695 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:21.519705 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:21.519712 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:21.520211 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:21.520403 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:21.521227 (+ 824us) server_negotiation.cc:299] Negotiation successful
0619 01:11:21.521595 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":230,"threads_started":1}
I20260619 01:11:21.522816 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:21.500910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.1:52811)
0619 01:11:21.501755 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:21.501796 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:21.503722 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:21.504592 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:21.504602 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:21.505187 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:21.505905 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:21.505925 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:21.507040 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:21.507044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:21.508402 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:21.508415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:21.509356 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:21.509365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:21.509627 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:21.510400 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:11:21.510430 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:21.512822 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:21.517897 (+ 5075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:21.517906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:21.517924 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:21.518627 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:21.519242 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:21.519245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:21.519248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:21.519544 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:21.520362 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:21.520368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:21.520736 (+ 368us) client_negotiation.cc:769] Sending connection context
0619 01:11:21.520986 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 01:11:21.521593 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":401,"threads_started":1}
I20260619 01:11:21.524546 17756 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35567
I20260619 01:11:21.524925 17756 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:21.525949 17756 heartbeater.cc:511] Master 127.0.97.62:35567 requested a full tablet report, sending...
I20260619 01:11:21.528529 17567 ts_manager.cc:194] Registered new tserver with Master: c1c0cdf0105743ffb70c2e8ee6e88ede (127.0.97.1:38779)
I20260619 01:11:21.530133 17567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52811
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:11:21.563989 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:21.906173 17762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:21.906571 17762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:21.906739 17762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:21.914331 17762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:21.914521 17762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:21.914721 17762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:11:21.924993 17762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:21.926443 17762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:21.927861 17762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:21.942518 17768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:21.943056 17770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:21.943111 17762 server_base.cc:1034] running on GCE node
W20260619 01:11:21.943241 17767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:21.944023 17762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:21.945143 17762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:21.946419 17762 hybrid_clock.cc:648] HybridClock initialized: now 1781831481946365 us; error 55 us; skew 500 ppm
I20260619 01:11:21.946802 17762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:21 dist-test-slave-5p8w krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:21.950295 17762 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:11:21.952075 17762 webserver.cc:460] Webserver started at http://127.0.97.2:45851/ using document root <none> and password file <none>
I20260619 01:11:21.952852 17762 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:21.952996 17762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:21.953413 17762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:21.956578 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b"
format_stamp: "Formatted at 2026-06-19 01:11:21 on dist-test-slave-5p8w"
server_key: "25514bee1807fea2b11ad9d5bbb31f06"
server_key_iv: "b5446c6ac3fc7db8a7c6cf631a219de5"
server_key_version: "encryptionkey@0"
I20260619 01:11:21.957502 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b"
format_stamp: "Formatted at 2026-06-19 01:11:21 on dist-test-slave-5p8w"
server_key: "25514bee1807fea2b11ad9d5bbb31f06"
server_key_iv: "b5446c6ac3fc7db8a7c6cf631a219de5"
server_key_version: "encryptionkey@0"
I20260619 01:11:21.963102 17762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 01:11:21.966954 17777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:21.968436 17762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:11:21.968622 17762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b"
format_stamp: "Formatted at 2026-06-19 01:11:21 on dist-test-slave-5p8w"
server_key: "25514bee1807fea2b11ad9d5bbb31f06"
server_key_iv: "b5446c6ac3fc7db8a7c6cf631a219de5"
server_key_version: "encryptionkey@0"
I20260619 01:11:21.968844 17762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:21.989516 17762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:21.993618 17762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:21.993939 17762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:21.994939 17762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:21.996330 17762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:21.996438 17762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:21.996611 17762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:21.996685 17762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:22.031235 17762 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36921
I20260619 01:11:22.031265 17889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36921 every 8 connection(s)
I20260619 01:11:22.032683 17762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:11:22.039319 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 17762
I20260619 01:11:22.039542 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:11:22.039992 388 external_mini_cluster.cc:1428] Setting key 0f7b61c4322dd4889b30f3ff9199352c
Jun 19 01:11:22 dist-test-slave-5p8w krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:22.056351 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.035634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34999 (local address 127.0.97.62:35567)
0619 01:11:22.036091 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:22.036095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:22.037084 (+ 989us) server_negotiation.cc:407] Connection header received
0619 01:11:22.039082 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:22.039088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:22.039185 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:22.039305 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:22.041137 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.042260 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.044884 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.045385 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.045492 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.050176 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:22.050181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:22.050184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:22.050220 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:22.053099 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:22.053943 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:22.053949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:22.053952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:22.054093 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:22.054547 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:22.054556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:22.054559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:22.054835 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:22.055000 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:22.055742 (+ 742us) server_negotiation.cc:299] Negotiation successful
0619 01:11:22.055974 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":253,"threads_started":1}
I20260619 01:11:22.057447 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.036185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.2:34999)
0619 01:11:22.036908 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:22.036955 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:22.038763 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:22.039606 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:22.039635 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:22.040254 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:22.040933 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.040957 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.042422 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.042427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.044715 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.044735 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.045629 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.045639 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.045984 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.046738 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:11:22.046778 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:22.049897 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:22.053286 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:22.053323 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:22.053342 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:22.053798 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:22.054219 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:22.054224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:22.054228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:22.054424 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:22.054974 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:22.054985 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:22.055360 (+ 375us) client_negotiation.cc:769] Sending connection context
0619 01:11:22.055639 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 01:11:22.056215 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":290,"threads_started":1}
I20260619 01:11:22.058984 17890 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35567
I20260619 01:11:22.059319 17890 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:22.060223 17890 heartbeater.cc:511] Master 127.0.97.62:35567 requested a full tablet report, sending...
I20260619 01:11:22.061939 17567 ts_manager.cc:194] Registered new tserver with Master: dfb958b1a21f4d51a7bd7a04b0f6dc0b (127.0.97.2:36921)
I20260619 01:11:22.063243 17567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34999
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:11:22.098920 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:22.440399 17896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:22.440706 17896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:22.440804 17896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:22.448426 17896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:22.448541 17896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:22.448654 17896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:11:22.458634 17896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:22.459904 17896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:22.461229 17896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:22.475168 17901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:22.475127 17902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:22.475092 17904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:22.476128 17896 server_base.cc:1034] running on GCE node
I20260619 01:11:22.476936 17896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:22.478184 17896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:22.479442 17896 hybrid_clock.cc:648] HybridClock initialized: now 1781831482479380 us; error 64 us; skew 500 ppm
I20260619 01:11:22.479782 17896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:22 dist-test-slave-5p8w krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:22.483201 17896 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:11:22.484822 17896 webserver.cc:460] Webserver started at http://127.0.97.3:35445/ using document root <none> and password file <none>
I20260619 01:11:22.485625 17896 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:22.485769 17896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:22.486090 17896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:22.489116 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "be206bd2d4af4943a9c1a9992e1864c4"
format_stamp: "Formatted at 2026-06-19 01:11:22 on dist-test-slave-5p8w"
server_key: "fb814da1c83c6c85707a369bdb869012"
server_key_iv: "60672867d8020d6aedca1879f5a0d7e1"
server_key_version: "encryptionkey@0"
I20260619 01:11:22.490032 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "be206bd2d4af4943a9c1a9992e1864c4"
format_stamp: "Formatted at 2026-06-19 01:11:22 on dist-test-slave-5p8w"
server_key: "fb814da1c83c6c85707a369bdb869012"
server_key_iv: "60672867d8020d6aedca1879f5a0d7e1"
server_key_version: "encryptionkey@0"
I20260619 01:11:22.495348 17896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:11:22.499033 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:22.500574 17896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:11:22.500746 17896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "be206bd2d4af4943a9c1a9992e1864c4"
format_stamp: "Formatted at 2026-06-19 01:11:22 on dist-test-slave-5p8w"
server_key: "fb814da1c83c6c85707a369bdb869012"
server_key_iv: "60672867d8020d6aedca1879f5a0d7e1"
server_key_version: "encryptionkey@0"
I20260619 01:11:22.500962 17896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:22.521065 17896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:22.526047 17896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:22.526366 17896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:22.527412 17896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:22.528788 17896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:22.528865 17896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:22.529062 17896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:22.529112 17896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:22.533566 17756 heartbeater.cc:503] Master 127.0.97.62:35567 was elected leader, sending a full tablet report...
I20260619 01:11:22.564092 17896 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34429
I20260619 01:11:22.564132 18023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34429 every 8 connection(s)
I20260619 01:11:22.565478 17896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:11:22.574219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 17896
I20260619 01:11:22.574432 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:11:22.574906 388 external_mini_cluster.cc:1428] Setting key d1ab678be21646af5a501cb1f1acba38
Jun 19 01:11:22 dist-test-slave-5p8w krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:22.586653 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.568036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55093 (local address 127.0.97.62:35567)
0619 01:11:22.568543 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:22.568550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:22.569577 (+ 1027us) server_negotiation.cc:407] Connection header received
0619 01:11:22.571401 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:22.571405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:22.571475 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:22.571572 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:22.573187 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.574294 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.576081 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.576545 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.576695 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.580447 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:22.580452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:22.580456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:22.580492 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:22.583068 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:22.584153 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:22.584157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:22.584160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:22.584361 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:22.584808 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:22.584813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:22.584816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:22.585107 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:22.585275 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:22.586027 (+ 752us) server_negotiation.cc:299] Negotiation successful
0619 01:11:22.586267 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":238,"threads_started":1}
I20260619 01:11:22.587683 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.568587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35567 (local address 127.0.97.3:55093)
0619 01:11:22.569404 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:22.569449 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:22.571135 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:22.571792 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:22.571800 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:22.572338 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:22.573015 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.573034 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.574494 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.574498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.575911 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.575926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.576813 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.576820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.577082 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.577849 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:11:22.577873 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:22.580204 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:22.583217 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:22.583227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:22.583246 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:22.584010 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:22.584483 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:22.584488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:22.584491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:22.584692 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:22.585239 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:22.585250 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:22.585648 (+ 398us) client_negotiation.cc:769] Sending connection context
0619 01:11:22.585937 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 01:11:22.586506 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":313,"threads_started":1}
I20260619 01:11:22.589401 18024 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35567
I20260619 01:11:22.589735 18024 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:22.590617 18024 heartbeater.cc:511] Master 127.0.97.62:35567 requested a full tablet report, sending...
I20260619 01:11:22.592056 17567 ts_manager.cc:194] Registered new tserver with Master: be206bd2d4af4943a9c1a9992e1864c4 (127.0.97.3:34429)
I20260619 01:11:22.592888 17567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55093
I20260619 01:11:22.603240 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:11:22.620664 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.607858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.0.97.62:35567)
0619 01:11:22.608027 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:22.608031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:22.608590 (+ 559us) server_negotiation.cc:407] Connection header received
0619 01:11:22.608732 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:22.608736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:22.608800 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:22.608888 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:22.610506 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.611245 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.612638 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.613033 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.613126 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.615700 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:22.615706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:22.615709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:22.615743 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:22.618231 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:22.618886 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:22.618892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:22.618895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:22.619027 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:22.619342 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:22.619345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:22.619347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:22.619665 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:22.619964 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:22.620233 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 01:11:22.620430 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:11:22.624956 17567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:11:22.632081 17567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:11:22.670293 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.645653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.0.97.2:36921)
0619 01:11:22.661889 (+ 16236us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:22.661901 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:22.662749 (+ 848us) server_negotiation.cc:407] Connection header received
0619 01:11:22.662966 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:22.662974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:22.663186 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:22.663393 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:22.664230 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.665621 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.667910 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.669074 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.669229 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.669498 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:22.669762 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 01:11:22.669931 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16004,"thread_start_us":16238,"threads_started":1}
I20260619 01:11:22.670368 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.645046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36921 (local address 127.0.0.1:33312)
0619 01:11:22.662625 (+ 17579us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:22.662643 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:22.662779 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:22.663641 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:22.663648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:22.663674 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:22.664045 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.664058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.665757 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.665763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.667717 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.667732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.669270 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.669276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.669500 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.669562 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 01:11:22.669718 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 01:11:22.669928 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17445,"mutex_wait_us":3,"thread_start_us":9952,"threads_started":1}
I20260619 01:11:22.671301 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.644163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34429 (local address 127.0.0.1:35254)
0619 01:11:22.655442 (+ 11279us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:22.655496 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:22.655711 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:22.662624 (+ 6913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:22.662629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:22.662665 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:22.663204 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.663214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.665151 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.665160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.668139 (+ 2979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.668149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.670399 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.670406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.670598 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.670632 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 01:11:22.670724 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 01:11:22.670870 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11017,"mutex_wait_us":190,"thread_start_us":18163,"threads_started":1}
I20260619 01:11:22.671414 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.644474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.0.97.3:34429)
0619 01:11:22.661885 (+ 17411us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:22.661894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:22.661932 (+ 38us) server_negotiation.cc:407] Connection header received
0619 01:11:22.662029 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:22.662037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:22.662263 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:22.662450 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:22.663451 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.665004 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.668323 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.669894 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.670025 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.670237 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:22.670915 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 01:11:22.671093 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17220,"thread_start_us":17779,"threads_started":1}
I20260619 01:11:22.672194 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.644556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38779 (local address 127.0.0.1:51976)
0619 01:11:22.660209 (+ 15653us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:22.660231 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:22.661101 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:22.662595 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:22.662605 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:22.662641 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:22.663149 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.663167 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.665215 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.665220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.668127 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.668142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.671304 (+ 3162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.671310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.671501 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.671532 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 01:11:22.671614 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 01:11:22.671755 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15509,"mutex_wait_us":207,"thread_start_us":15363,"threads_started":1}
I20260619 01:11:22.672963 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.645120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.0.97.1:38779)
0619 01:11:22.653988 (+ 8868us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:22.654002 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:22.660382 (+ 6380us) server_negotiation.cc:407] Connection header received
0619 01:11:22.661539 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:22.661552 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:22.661877 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:22.662068 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:22.663424 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.665077 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.669267 (+ 4190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.670767 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.670877 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.671121 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:22.672392 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0619 01:11:22.672570 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8608,"thread_start_us":8400,"threads_started":1}
I20260619 01:11:22.674980 17825 tablet_service.cc:1467] Processing CreateTablet for tablet d9fdbd016e6e4e038bb26c241da92826 (DEFAULT_TABLE table=test-table [id=d5e076af9d1c40fe949f669f83f6cdfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:22.675797 17959 tablet_service.cc:1467] Processing CreateTablet for tablet d9fdbd016e6e4e038bb26c241da92826 (DEFAULT_TABLE table=test-table [id=d5e076af9d1c40fe949f669f83f6cdfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:22.676537 17825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fdbd016e6e4e038bb26c241da92826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:22.677209 17959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fdbd016e6e4e038bb26c241da92826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:22.684719 17691 tablet_service.cc:1467] Processing CreateTablet for tablet d9fdbd016e6e4e038bb26c241da92826 (DEFAULT_TABLE table=test-table [id=d5e076af9d1c40fe949f669f83f6cdfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:22.686023 17691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9fdbd016e6e4e038bb26c241da92826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:22.692296 18043 tablet_bootstrap.cc:492] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: Bootstrap starting.
I20260619 01:11:22.693145 18044 tablet_bootstrap.cc:492] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: Bootstrap starting.
I20260619 01:11:22.696506 18043 tablet_bootstrap.cc:654] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:22.696979 18044 tablet_bootstrap.cc:654] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:22.697733 18043 log.cc:824] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:22.698246 18044 log.cc:824] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:22.700198 18047 tablet_bootstrap.cc:492] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: Bootstrap starting.
I20260619 01:11:22.701000 18043 tablet_bootstrap.cc:492] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: No bootstrap required, opened a new log
I20260619 01:11:22.701272 18043 ts_tablet_manager.cc:1397] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 01:11:22.702633 18044 tablet_bootstrap.cc:492] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: No bootstrap required, opened a new log
I20260619 01:11:22.702926 18044 ts_tablet_manager.cc:1397] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260619 01:11:22.703989 18047 tablet_bootstrap.cc:654] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:22.705245 18047 log.cc:824] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:22.708387 18047 tablet_bootstrap.cc:492] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: No bootstrap required, opened a new log
I20260619 01:11:22.708683 18047 ts_tablet_manager.cc:1397] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 01:11:22.711862 18044 raft_consensus.cc:348] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be206bd2d4af4943a9c1a9992e1864c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34429 } } peers { permanent_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38779 } } peers { permanent_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36921 } }
I20260619 01:11:22.711858 18043 raft_consensus.cc:348] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be206bd2d4af4943a9c1a9992e1864c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34429 } } peers { permanent_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38779 } } peers { permanent_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36921 } }
I20260619 01:11:22.712327 18044 raft_consensus.cc:374] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:22.712337 18043 raft_consensus.cc:374] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:22.712438 18043 raft_consensus.cc:729] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb958b1a21f4d51a7bd7a04b0f6dc0b, State: Initialized, Role: FOLLOWER
I20260619 01:11:22.712498 18044 raft_consensus.cc:729] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be206bd2d4af4943a9c1a9992e1864c4, State: Initialized, Role: FOLLOWER
I20260619 01:11:22.713128 18044 consensus_queue.cc:260] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be206bd2d4af4943a9c1a9992e1864c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34429 } } peers { permanent_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38779 } } peers { permanent_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36921 } }
I20260619 01:11:22.713116 18043 consensus_queue.cc:260] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be206bd2d4af4943a9c1a9992e1864c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34429 } } peers { permanent_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38779 } } peers { permanent_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36921 } }
I20260619 01:11:22.714340 18024 heartbeater.cc:503] Master 127.0.97.62:35567 was elected leader, sending a full tablet report...
I20260619 01:11:22.715046 18044 ts_tablet_manager.cc:1428] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260619 01:11:22.716085 17890 heartbeater.cc:503] Master 127.0.97.62:35567 was elected leader, sending a full tablet report...
I20260619 01:11:22.718130 18043 ts_tablet_manager.cc:1428] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: Time spent starting tablet: real 0.017s user 0.006s sys 0.009s
I20260619 01:11:22.720118 18047 raft_consensus.cc:348] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be206bd2d4af4943a9c1a9992e1864c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34429 } } peers { permanent_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38779 } } peers { permanent_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36921 } }
I20260619 01:11:22.720628 18047 raft_consensus.cc:374] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:22.720752 18047 raft_consensus.cc:729] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c0cdf0105743ffb70c2e8ee6e88ede, State: Initialized, Role: FOLLOWER
I20260619 01:11:22.721545 18047 consensus_queue.cc:260] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be206bd2d4af4943a9c1a9992e1864c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34429 } } peers { permanent_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38779 } } peers { permanent_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36921 } }
I20260619 01:11:22.723245 18047 ts_tablet_manager.cc:1428] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260619 01:11:22.749750 17757 tablet.cc:2369] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:11:22.766790 18051 raft_consensus.cc:482] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:11:22.767138 18051 raft_consensus.cc:504] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be206bd2d4af4943a9c1a9992e1864c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34429 } } peers { permanent_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38779 } } peers { permanent_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36921 } }
I20260619 01:11:22.768875 18051 leader_election.cc:288] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be206bd2d4af4943a9c1a9992e1864c4 (127.0.97.3:34429), dfb958b1a21f4d51a7bd7a04b0f6dc0b (127.0.97.2:36921)
I20260619 01:11:22.776765 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.769323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48639 (local address 127.0.97.2:36921)
0619 01:11:22.769534 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:22.769540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:22.770278 (+ 738us) server_negotiation.cc:407] Connection header received
0619 01:11:22.770486 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:22.770492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:22.770572 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:22.770675 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:22.771573 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.772723 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.774994 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.775779 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.775948 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.776062 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:22.776367 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 01:11:22.776498 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:11:22.776798 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.769302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52789 (local address 127.0.97.3:34429)
0619 01:11:22.769534 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:22.769540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:22.770279 (+ 739us) server_negotiation.cc:407] Connection header received
0619 01:11:22.770487 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:22.770492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:22.770572 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:22.770675 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:22.771573 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.772722 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.775007 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.775786 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.775951 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.776079 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:22.776394 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 01:11:22.776529 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 01:11:22.776968 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.769178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36921 (local address 127.0.97.1:48639)
0619 01:11:22.770139 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:22.770160 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:22.770316 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:22.770706 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:22.770710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:22.770749 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:22.771415 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.771424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.772879 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.772892 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.774817 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.774832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.776037 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.776043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.776199 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.776223 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 01:11:22.776333 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 01:11:22.776472 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":123,"thread_start_us":542,"threads_started":1}
I20260619 01:11:22.777158 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.769246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34429 (local address 127.0.97.1:52789)
0619 01:11:22.770139 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:22.770160 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:22.770345 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:22.770706 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:22.770710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:22.770753 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:22.771379 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.771411 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.772880 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.772884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.774827 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:22.774842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.776079 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.776086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:22.776238 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.776260 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:11:22.776365 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:11:22.776508 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":188,"spinlock_wait_cycles":60416,"thread_start_us":500,"threads_started":1}
I20260619 01:11:22.778260 17979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fdbd016e6e4e038bb26c241da92826" candidate_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be206bd2d4af4943a9c1a9992e1864c4" is_pre_election: true
I20260619 01:11:22.778257 17845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fdbd016e6e4e038bb26c241da92826" candidate_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" is_pre_election: true
I20260619 01:11:22.778880 17845 raft_consensus.cc:2454] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1c0cdf0105743ffb70c2e8ee6e88ede in term 0.
I20260619 01:11:22.778880 17979 raft_consensus.cc:2454] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1c0cdf0105743ffb70c2e8ee6e88ede in term 0.
I20260619 01:11:22.779700 17646 leader_election.cc:302] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: be206bd2d4af4943a9c1a9992e1864c4, c1c0cdf0105743ffb70c2e8ee6e88ede, dfb958b1a21f4d51a7bd7a04b0f6dc0b; no voters:
I20260619 01:11:22.780177 18051 raft_consensus.cc:2781] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:11:22.780336 18051 raft_consensus.cc:482] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:11:22.780458 18051 raft_consensus.cc:3037] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:22.782874 18051 raft_consensus.cc:504] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be206bd2d4af4943a9c1a9992e1864c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34429 } } peers { permanent_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38779 } } peers { permanent_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36921 } }
I20260619 01:11:22.783581 18051 leader_election.cc:288] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [CANDIDATE]: Term 1 election: Requested vote from peers be206bd2d4af4943a9c1a9992e1864c4 (127.0.97.3:34429), dfb958b1a21f4d51a7bd7a04b0f6dc0b (127.0.97.2:36921)
I20260619 01:11:22.784161 17979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fdbd016e6e4e038bb26c241da92826" candidate_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be206bd2d4af4943a9c1a9992e1864c4"
I20260619 01:11:22.784199 17845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9fdbd016e6e4e038bb26c241da92826" candidate_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b"
I20260619 01:11:22.784409 17979 raft_consensus.cc:3037] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:22.784432 17845 raft_consensus.cc:3037] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:22.786939 17979 raft_consensus.cc:2454] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1c0cdf0105743ffb70c2e8ee6e88ede in term 1.
I20260619 01:11:22.786942 17845 raft_consensus.cc:2454] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1c0cdf0105743ffb70c2e8ee6e88ede in term 1.
W20260619 01:11:22.787477 17891 tablet.cc:2369] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:11:22.787469 17646 leader_election.cc:302] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be206bd2d4af4943a9c1a9992e1864c4, c1c0cdf0105743ffb70c2e8ee6e88ede; no voters:
I20260619 01:11:22.787797 18051 raft_consensus.cc:2781] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:22.788379 18051 raft_consensus.cc:686] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 1 LEADER]: Becoming Leader. State: Replica: c1c0cdf0105743ffb70c2e8ee6e88ede, State: Running, Role: LEADER
I20260619 01:11:22.788942 18051 consensus_queue.cc:237] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be206bd2d4af4943a9c1a9992e1864c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34429 } } peers { permanent_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38779 } } peers { permanent_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36921 } }
I20260619 01:11:22.793993 17566 catalog_manager.cc:5495] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede reported cstate change: term changed from 0 to 1, leader changed from <none> to c1c0cdf0105743ffb70c2e8ee6e88ede (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be206bd2d4af4943a9c1a9992e1864c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1c0cdf0105743ffb70c2e8ee6e88ede" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36921 } health_report { overall_health: UNKNOWN } } }
W20260619 01:11:22.817190 18025 tablet.cc:2369] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:11:22.827046 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.821778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.0.97.1:38779)
0619 01:11:22.821962 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:22.821967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:22.822018 (+ 51us) server_negotiation.cc:407] Connection header received
0619 01:11:22.822198 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:22.822201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:22.822281 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:22.822422 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:11:22.822755 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.823488 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.825008 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.825470 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.825564 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.826050 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:11:22.826597 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:11:22.826703 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:22.826777 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 01:11:22.826841 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 01:11:22.835479 17845 raft_consensus.cc:1261] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b [term 1 FOLLOWER]: Refusing update from remote peer c1c0cdf0105743ffb70c2e8ee6e88ede: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:22.835480 17979 raft_consensus.cc:1261] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4 [term 1 FOLLOWER]: Refusing update from remote peer c1c0cdf0105743ffb70c2e8ee6e88ede: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:22.836578 18054 consensus_queue.cc:1035] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb958b1a21f4d51a7bd7a04b0f6dc0b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:22.836926 18051 consensus_queue.cc:1035] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "be206bd2d4af4943a9c1a9992e1864c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:22.849908 18057 mvcc.cc:205] Tried to move back new op lower bound from 7298381753683312640 to 7298381753508188160. Current Snapshot: MvccSnapshot[applied={T|T < 7298381753683312640}]
I20260619 01:11:22.852578 18058 mvcc.cc:205] Tried to move back new op lower bound from 7298381753683312640 to 7298381753508188160. Current Snapshot: MvccSnapshot[applied={T|T < 7298381753683312640}]
I20260619 01:11:22.865164 17566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:22.865511 17566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:22.868881 17566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1e146337da54293b3d19f0067686e98: Sending DeleteTablet for 3 replicas of tablet d9fdbd016e6e4e038bb26c241da92826
I20260619 01:11:22.870101 17959 tablet_service.cc:1514] Processing DeleteTablet for tablet d9fdbd016e6e4e038bb26c241da92826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35254
I20260619 01:11:22.870383 17825 tablet_service.cc:1514] Processing DeleteTablet for tablet d9fdbd016e6e4e038bb26c241da92826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33312
I20260619 01:11:22.870880 17691 tablet_service.cc:1514] Processing DeleteTablet for tablet d9fdbd016e6e4e038bb26c241da92826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51976
I20260619 01:11:22.871887 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 17628
I20260619 01:11:22.887665 18069 tablet_replica.cc:330] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: stopping tablet replica
I20260619 01:11:22.888453 18069 raft_consensus.cc:2229] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:11:22.889436 18069 raft_consensus.cc:2258] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:22.889719 18067 tablet_replica.cc:330] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: stopping tablet replica
I20260619 01:11:22.890189 18067 raft_consensus.cc:2229] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:22.890722 18067 raft_consensus.cc:2258] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:22.893011 18067 ts_tablet_manager.cc:1905] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:22.895159 18069 ts_tablet_manager.cc:1905] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:22.900456 18067 ts_tablet_manager.cc:1918] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:22.900740 18067 log.cc:1196] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/d9fdbd016e6e4e038bb26c241da92826
I20260619 01:11:22.901386 18067 ts_tablet_manager.cc:1939] T d9fdbd016e6e4e038bb26c241da92826 P be206bd2d4af4943a9c1a9992e1864c4: Deleting consensus metadata
I20260619 01:11:22.903575 17553 catalog_manager.cc:4841] TS be206bd2d4af4943a9c1a9992e1864c4 (127.0.97.3:34429): tablet d9fdbd016e6e4e038bb26c241da92826 (table test-table [id=d5e076af9d1c40fe949f669f83f6cdfc]) successfully deleted
I20260619 01:11:22.905036 18069 ts_tablet_manager.cc:1918] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:22.905236 18069 log.cc:1196] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/d9fdbd016e6e4e038bb26c241da92826
I20260619 01:11:22.908084 18068 tablet_replica.cc:330] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: stopping tablet replica
Jun 19 01:11:22 dist-test-slave-5p8w krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:11:22.908490 18069 ts_tablet_manager.cc:1939] T d9fdbd016e6e4e038bb26c241da92826 P c1c0cdf0105743ffb70c2e8ee6e88ede: Deleting consensus metadata
I20260619 01:11:22.908586 18068 raft_consensus.cc:2229] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:22.909096 18068 raft_consensus.cc:2258] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:22.910539 17554 catalog_manager.cc:4841] TS c1c0cdf0105743ffb70c2e8ee6e88ede (127.0.97.1:38779): tablet d9fdbd016e6e4e038bb26c241da92826 (table test-table [id=d5e076af9d1c40fe949f669f83f6cdfc]) successfully deleted
I20260619 01:11:22.912256 18068 ts_tablet_manager.cc:1905] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:22.917733 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:22.873529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.0.97.1:38779)
0619 01:11:22.873771 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:22.873779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:22.883976 (+ 10197us) server_negotiation.cc:407] Connection header received
0619 01:11:22.884553 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:22.884563 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:22.884672 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:22.884767 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:22.889704 (+ 4937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.890939 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.895857 (+ 4918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:22.896533 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:22.896659 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:22.910612 (+ 13953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:22.910628 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:22.910646 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:22.910705 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:22.914438 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:22.915271 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:22.915288 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:22.915290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:22.915476 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:22.915921 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:22.915930 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:22.915933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:22.916304 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:22.916654 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:22.916966 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 01:11:22.917211 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:11:22.918573 17751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:22.919617 18068 ts_tablet_manager.cc:1918] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:22.919829 18068 log.cc:1196] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/d9fdbd016e6e4e038bb26c241da92826
I20260619 01:11:22.920426 18068 ts_tablet_manager.cc:1939] T d9fdbd016e6e4e038bb26c241da92826 P dfb958b1a21f4d51a7bd7a04b0f6dc0b: Deleting consensus metadata
I20260619 01:11:22.922531 17552 catalog_manager.cc:4841] TS dfb958b1a21f4d51a7bd7a04b0f6dc0b (127.0.97.2:36921): tablet d9fdbd016e6e4e038bb26c241da92826 (table test-table [id=d5e076af9d1c40fe949f669f83f6cdfc]) successfully deleted
I20260619 01:11:23.020215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 17628
I20260619 01:11:23.041981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 17762
Jun 19 01:11:23 dist-test-slave-5p8w krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:11:23.058151 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:23.042802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.0.97.2:36921)
0619 01:11:23.043036 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:23.043042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:23.043062 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:11:23.043203 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:23.043208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:23.043285 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:23.043394 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:23.044674 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:23.045625 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:23.046962 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:23.047435 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:23.047569 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:23.052156 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:23.052168 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:23.052181 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:23.052225 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:23.055227 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:23.055970 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:23.055983 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:23.055985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:23.056111 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:23.056505 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:23.056511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:23.056514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:23.056912 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:23.057222 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:23.057574 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 01:11:23.057805 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 01:11:23.058873 17885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:23.152346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 17762
I20260619 01:11:23.169705 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 17896
Jun 19 01:11:23 dist-test-slave-5p8w krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:11:23.185964 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:23.170640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.0.97.3:34429)
0619 01:11:23.170814 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:23.170818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:23.170838 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:11:23.171068 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:23.171073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:23.171154 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:23.171245 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:23.172588 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:23.173509 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:23.174966 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:23.175366 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:23.175472 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:23.180012 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:23.180031 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:23.180048 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:23.180089 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:23.183169 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:23.183912 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:23.183920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:23.183922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:23.184068 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:23.184385 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:23.184388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:23.184390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:23.184739 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:23.184959 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:23.185228 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 01:11:23.185650 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 01:11:23.186798 18019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:23.277391 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 17896
I20260619 01:11:23.295008 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 17535
I20260619 01:11:23.296141 17597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:23.410025 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 17535
2026-06-19T01:11:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4463 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T01:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:23Z Disabled control of system clock
I20260619 01:11:23.492565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46829
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:11:23.831046 18078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:23.831372 18078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:23.831456 18078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:23.839061 18078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:23.839185 18078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:23.839231 18078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:23.839289 18078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:23.839334 18078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:23.848728 18078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46829
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:23.850067 18078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:23.851831 18078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:23.864678 18083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:23.864651 18084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:23.864605 18086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:23.865404 18078 server_base.cc:1034] running on GCE node
I20260619 01:11:23.866359 18078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:23.867904 18078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:23.869150 18078 hybrid_clock.cc:648] HybridClock initialized: now 1781831483869099 us; error 48 us; skew 500 ppm
I20260619 01:11:23.869560 18078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:23.876284 18078 webserver.cc:460] Webserver started at http://127.0.97.62:37517/ using document root <none> and password file <none>
I20260619 01:11:23.877108 18078 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:23.877257 18078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:23.877640 18078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:23.880781 18078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "b4eb5e20ff2c4c0e86d9fa2f91ffcde7"
format_stamp: "Formatted at 2026-06-19 01:11:23 on dist-test-slave-5p8w"
server_key: "dd9fd85cd55e0bf3e21823b353f0b516"
server_key_iv: "ce144c845db3d5af54c3f282a8c8fac9"
server_key_version: "encryptionkey@0"
I20260619 01:11:23.881704 18078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "b4eb5e20ff2c4c0e86d9fa2f91ffcde7"
format_stamp: "Formatted at 2026-06-19 01:11:23 on dist-test-slave-5p8w"
server_key: "dd9fd85cd55e0bf3e21823b353f0b516"
server_key_iv: "ce144c845db3d5af54c3f282a8c8fac9"
server_key_version: "encryptionkey@0"
I20260619 01:11:23.887247 18078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:11:23.890771 18092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:23.892242 18078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:11:23.892439 18078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "b4eb5e20ff2c4c0e86d9fa2f91ffcde7"
format_stamp: "Formatted at 2026-06-19 01:11:23 on dist-test-slave-5p8w"
server_key: "dd9fd85cd55e0bf3e21823b353f0b516"
server_key_iv: "ce144c845db3d5af54c3f282a8c8fac9"
server_key_version: "encryptionkey@0"
I20260619 01:11:23.892665 18078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:23.908864 18078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:23.909842 18078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:23.910156 18078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:23.929278 18078 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46829
I20260619 01:11:23.929355 18143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46829 every 8 connection(s)
I20260619 01:11:23.930815 18078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:23.934911 18144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:23.937146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 18078
I20260619 01:11:23.937358 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:23.937901 388 external_mini_cluster.cc:1428] Setting key f7b5f276ff7421d9c832099979da9f3c
I20260619 01:11:23.947276 18144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7: Bootstrap starting.
I20260619 01:11:23.950542 18144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:23.951730 18144 log.cc:824] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:23.951844 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:23.939917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.0.97.62:46829)
0619 01:11:23.941008 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:23.941020 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:23.941065 (+ 45us) server_negotiation.cc:407] Connection header received
0619 01:11:23.942869 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:23.942893 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:23.943384 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:23.943778 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:23.944386 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:23.945777 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:23.947191 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:23.947954 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:23.948155 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:23.949435 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:23.949444 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:23.949459 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:23.950008 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:23.950101 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:23.950127 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:23.950255 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:23.950514 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 01:11:23.950845 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":433,"threads_started":1}
I20260619 01:11:23.954916 18144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7: No bootstrap required, opened a new log
I20260619 01:11:23.962809 18144 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb5e20ff2c4c0e86d9fa2f91ffcde7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46829 } }
I20260619 01:11:23.963256 18144 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:23.963374 18144 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4eb5e20ff2c4c0e86d9fa2f91ffcde7, State: Initialized, Role: FOLLOWER
I20260619 01:11:23.964056 18144 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb5e20ff2c4c0e86d9fa2f91ffcde7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46829 } }
I20260619 01:11:23.964316 18144 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:23.964466 18144 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:23.964660 18144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:23.967036 18144 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb5e20ff2c4c0e86d9fa2f91ffcde7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46829 } }
I20260619 01:11:23.967574 18144 leader_election.cc:302] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4eb5e20ff2c4c0e86d9fa2f91ffcde7; no voters:
I20260619 01:11:23.968185 18144 leader_election.cc:288] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:23.968341 18149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:23.969076 18149 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [term 1 LEADER]: Becoming Leader. State: Replica: b4eb5e20ff2c4c0e86d9fa2f91ffcde7, State: Running, Role: LEADER
I20260619 01:11:23.969898 18149 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb5e20ff2c4c0e86d9fa2f91ffcde7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46829 } }
I20260619 01:11:23.970408 18144 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:23.972734 18151 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4eb5e20ff2c4c0e86d9fa2f91ffcde7. Latest consensus state: current_term: 1 leader_uuid: "b4eb5e20ff2c4c0e86d9fa2f91ffcde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb5e20ff2c4c0e86d9fa2f91ffcde7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46829 } } }
I20260619 01:11:23.973076 18151 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:23.973335 18150 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4eb5e20ff2c4c0e86d9fa2f91ffcde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb5e20ff2c4c0e86d9fa2f91ffcde7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46829 } } }
I20260619 01:11:23.973668 18150 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:23.977914 18155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:23.984035 18155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:23.993854 18155 catalog_manager.cc:1347] Generated new cluster ID: 28e213c87e564ba3be89db35bce19340
I20260619 01:11:23.994000 18155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:24.019953 18155 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:24.021339 18155 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:24.030900 18155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7: Generated new TSK 0
I20260619 01:11:24.031854 18155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:24.036394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46829
--builtin_ntp_servers=127.0.97.20:38889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:11:24.410719 18168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:24.411015 18168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:24.411103 18168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:24.418743 18168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:24.418854 18168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:24.418963 18168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:11:24.428134 18168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:24.429441 18168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:24.431159 18168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:24.444556 18173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:24.444597 18174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:24.444741 18176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:24.445238 18168 server_base.cc:1034] running on GCE node
I20260619 01:11:24.445967 18168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:24.446955 18168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:24.448195 18168 hybrid_clock.cc:648] HybridClock initialized: now 1781831484448117 us; error 73 us; skew 500 ppm
I20260619 01:11:24.448551 18168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:24.451050 18168 webserver.cc:460] Webserver started at http://127.0.97.1:42283/ using document root <none> and password file <none>
I20260619 01:11:24.451956 18168 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:24.452080 18168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:24.452499 18168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:24.455904 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "55fc89253eaf431bbff17fb36231bf85"
format_stamp: "Formatted at 2026-06-19 01:11:24 on dist-test-slave-5p8w"
server_key: "7f911764d36be60cfac7e34fb6da5e61"
server_key_iv: "8905d8034ca6e0671f14b058b1893ec2"
server_key_version: "encryptionkey@0"
I20260619 01:11:24.456668 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "55fc89253eaf431bbff17fb36231bf85"
format_stamp: "Formatted at 2026-06-19 01:11:24 on dist-test-slave-5p8w"
server_key: "7f911764d36be60cfac7e34fb6da5e61"
server_key_iv: "8905d8034ca6e0671f14b058b1893ec2"
server_key_version: "encryptionkey@0"
I20260619 01:11:24.461892 18168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 01:11:24.465430 18182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:24.466823 18168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:11:24.466991 18168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "55fc89253eaf431bbff17fb36231bf85"
format_stamp: "Formatted at 2026-06-19 01:11:24 on dist-test-slave-5p8w"
server_key: "7f911764d36be60cfac7e34fb6da5e61"
server_key_iv: "8905d8034ca6e0671f14b058b1893ec2"
server_key_version: "encryptionkey@0"
I20260619 01:11:24.467221 18168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:24.484582 18168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:24.485567 18168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:24.485929 18168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:24.486999 18168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:24.488587 18168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:24.488679 18168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:24.488759 18168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:24.488814 18168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:24.522307 18168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34425
I20260619 01:11:24.522356 18294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34425 every 8 connection(s)
I20260619 01:11:24.523640 18168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:11:24.528880 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 18168
I20260619 01:11:24.529078 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:11:24.529561 388 external_mini_cluster.cc:1428] Setting key 55bb3d4ef941cc26d0edc9659cf0744b
I20260619 01:11:24.532601 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46829
--builtin_ntp_servers=127.0.97.20:38889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:11:24.536888 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:24.526292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51659 (local address 127.0.97.62:46829)
0619 01:11:24.526753 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:24.526758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:24.527844 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 01:11:24.529921 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:24.529925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:24.530002 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:24.530088 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:24.531486 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:24.532232 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:24.534137 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:24.534540 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:24.534675 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:24.536050 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:24.536055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:24.536058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:24.536077 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:24.536122 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:24.536134 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:24.536229 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:24.536450 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 01:11:24.536547 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":248,"threads_started":1}
I20260619 01:11:24.538079 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:24.526776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46829 (local address 127.0.97.1:51659)
0619 01:11:24.527680 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:24.527717 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:24.529527 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:24.530365 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:24.530373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:24.530475 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:24.531256 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:24.531283 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:24.532383 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:24.532395 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:24.533984 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:24.533996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:24.534762 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:24.534769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:24.535011 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:24.535715 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:11:24.535718 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:24.535758 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:11:24.535837 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:24.536228 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:24.536241 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:24.536251 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 01:11:24.536440 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 01:11:24.536854 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":281,"threads_started":1}
I20260619 01:11:24.539739 18295 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46829
I20260619 01:11:24.540107 18295 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:24.541106 18295 heartbeater.cc:511] Master 127.0.97.62:46829 requested a full tablet report, sending...
I20260619 01:11:24.543469 18109 ts_manager.cc:194] Registered new tserver with Master: 55fc89253eaf431bbff17fb36231bf85 (127.0.97.1:34425)
I20260619 01:11:24.545665 18109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:51659
W20260619 01:11:24.867976 18299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:24.868294 18299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:24.868386 18299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:24.876717 18299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:24.876837 18299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:24.876951 18299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:11:24.886586 18299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:24.887903 18299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:24.889765 18299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:24.903627 18304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:24.903700 18305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:24.903652 18307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:24.904492 18299 server_base.cc:1034] running on GCE node
I20260619 01:11:24.905095 18299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:24.906155 18299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:24.907390 18299 hybrid_clock.cc:648] HybridClock initialized: now 1781831484907334 us; error 47 us; skew 500 ppm
I20260619 01:11:24.907769 18299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:24.910359 18299 webserver.cc:460] Webserver started at http://127.0.97.2:39787/ using document root <none> and password file <none>
I20260619 01:11:24.911126 18299 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:24.911245 18299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:24.911595 18299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:24.914842 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "af6114ad9b9d4a59bdb2da68165e5f18"
format_stamp: "Formatted at 2026-06-19 01:11:24 on dist-test-slave-5p8w"
server_key: "b3a7153c3567fc12794b4cf38eb5e4a6"
server_key_iv: "2c1932d11d46db0137d5929493f352d7"
server_key_version: "encryptionkey@0"
I20260619 01:11:24.915707 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "af6114ad9b9d4a59bdb2da68165e5f18"
format_stamp: "Formatted at 2026-06-19 01:11:24 on dist-test-slave-5p8w"
server_key: "b3a7153c3567fc12794b4cf38eb5e4a6"
server_key_iv: "2c1932d11d46db0137d5929493f352d7"
server_key_version: "encryptionkey@0"
I20260619 01:11:24.921415 18299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 01:11:24.925153 18313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:24.926661 18299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:24.926855 18299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "af6114ad9b9d4a59bdb2da68165e5f18"
format_stamp: "Formatted at 2026-06-19 01:11:24 on dist-test-slave-5p8w"
server_key: "b3a7153c3567fc12794b4cf38eb5e4a6"
server_key_iv: "2c1932d11d46db0137d5929493f352d7"
server_key_version: "encryptionkey@0"
I20260619 01:11:24.927119 18299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:24.946241 18299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:24.947110 18299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:24.947424 18299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:24.948432 18299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:24.950022 18299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:24.950145 18299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:24.950263 18299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:24.950349 18299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:24.983592 18299 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41827
I20260619 01:11:24.983645 18425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41827 every 8 connection(s)
I20260619 01:11:24.984956 18299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:11:24.990603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 18299
I20260619 01:11:24.990780 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:11:24.991227 388 external_mini_cluster.cc:1428] Setting key 998d3f161f4dd638536166d9a49fce8c
I20260619 01:11:24.994424 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46829
--builtin_ntp_servers=127.0.97.20:38889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:11:24.999125 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:24.987624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33753 (local address 127.0.97.62:46829)
0619 01:11:24.987812 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:24.987817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:24.989212 (+ 1395us) server_negotiation.cc:407] Connection header received
0619 01:11:24.991173 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:24.991181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:24.991273 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:24.991398 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:24.992594 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:24.993612 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:24.996159 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:24.996569 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:24.996667 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:24.998384 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:24.998389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:24.998393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:24.998419 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:24.998451 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:24.998464 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:24.998536 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:24.998803 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 01:11:24.998887 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 01:11:25.000744 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:24.988129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46829 (local address 127.0.97.2:33753)
0619 01:11:24.989034 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:24.989077 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:24.990838 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:24.991663 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:24.991675 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:24.991795 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:24.992422 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:24.992439 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:24.993761 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:24.993767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:24.996009 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:24.996028 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:24.996812 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:24.996822 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:24.997169 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:24.998002 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:11:24.998007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:24.998064 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:11:24.998179 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:24.998568 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:24.998580 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:24.998592 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 01:11:24.998788 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 01:11:24.999250 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":274,"threads_started":1}
I20260619 01:11:25.002301 18426 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46829
I20260619 01:11:25.002638 18426 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:25.003602 18426 heartbeater.cc:511] Master 127.0.97.62:46829 requested a full tablet report, sending...
I20260619 01:11:25.005246 18109 ts_manager.cc:194] Registered new tserver with Master: af6114ad9b9d4a59bdb2da68165e5f18 (127.0.97.2:41827)
I20260619 01:11:25.006127 18109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:33753
W20260619 01:11:25.337037 18429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:25.337412 18429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:25.337579 18429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:25.345148 18429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:25.345265 18429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:25.345510 18429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:11:25.355043 18429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:25.356410 18429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:25.358331 18429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:25.371474 18435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:25.371634 18434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:25.372597 18437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:25.372901 18429 server_base.cc:1034] running on GCE node
I20260619 01:11:25.373715 18429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:25.374966 18429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:25.376228 18429 hybrid_clock.cc:648] HybridClock initialized: now 1781831485376165 us; error 48 us; skew 500 ppm
I20260619 01:11:25.376693 18429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:25.379472 18429 webserver.cc:460] Webserver started at http://127.0.97.3:38717/ using document root <none> and password file <none>
I20260619 01:11:25.380280 18429 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:25.380410 18429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:25.380800 18429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:25.383977 18429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "85e9d82d35884b5a84c24957bbbfa1a7"
format_stamp: "Formatted at 2026-06-19 01:11:25 on dist-test-slave-5p8w"
server_key: "aecb00ff90a5fd2ce195940d228703da"
server_key_iv: "c8505a07cc1739c780c7378337cf57a1"
server_key_version: "encryptionkey@0"
I20260619 01:11:25.384845 18429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "85e9d82d35884b5a84c24957bbbfa1a7"
format_stamp: "Formatted at 2026-06-19 01:11:25 on dist-test-slave-5p8w"
server_key: "aecb00ff90a5fd2ce195940d228703da"
server_key_iv: "c8505a07cc1739c780c7378337cf57a1"
server_key_version: "encryptionkey@0"
I20260619 01:11:25.390410 18429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 01:11:25.394193 18443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:25.395557 18429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:11:25.395753 18429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "85e9d82d35884b5a84c24957bbbfa1a7"
format_stamp: "Formatted at 2026-06-19 01:11:25 on dist-test-slave-5p8w"
server_key: "aecb00ff90a5fd2ce195940d228703da"
server_key_iv: "c8505a07cc1739c780c7378337cf57a1"
server_key_version: "encryptionkey@0"
I20260619 01:11:25.396023 18429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:25.424189 18429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:25.425094 18429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:25.425447 18429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:25.426431 18429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:25.427930 18429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:25.428054 18429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:25.428196 18429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:25.428285 18429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:25.461239 18429 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37177
I20260619 01:11:25.461349 18555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37177 every 8 connection(s)
I20260619 01:11:25.462639 18429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:11:25.463078 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 18429
I20260619 01:11:25.463253 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:11:25.463840 388 external_mini_cluster.cc:1428] Setting key 84e12ad5ba8fd706cbbfbe2708ad29f0
I20260619 01:11:25.475422 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.465368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48093 (local address 127.0.97.62:46829)
0619 01:11:25.465545 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:25.465568 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:25.467465 (+ 1897us) server_negotiation.cc:407] Connection header received
0619 01:11:25.469213 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:25.469216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:25.469282 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:25.469412 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:25.470454 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.471193 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.472889 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.473384 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.473513 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.474753 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:25.474757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:25.474761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:25.474778 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:25.474810 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:25.474822 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:25.474892 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:25.475091 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 01:11:25.475168 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:11:25.476413 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.466059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46829 (local address 127.0.97.3:48093)
0619 01:11:25.467304 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:25.467356 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:25.468985 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:25.469609 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:25.469617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:25.469704 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:25.470282 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.470297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.471311 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.471315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.472764 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.472776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.473610 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.473616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.473813 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.474454 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:11:25.474456 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:25.474498 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:11:25.474572 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:25.474906 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:25.474912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:25.474918 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 01:11:25.475061 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 01:11:25.475341 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":513,"threads_started":1}
I20260619 01:11:25.478120 18556 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46829
I20260619 01:11:25.478508 18556 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:25.479468 18556 heartbeater.cc:511] Master 127.0.97.62:46829 requested a full tablet report, sending...
I20260619 01:11:25.480804 18109 ts_manager.cc:194] Registered new tserver with Master: 85e9d82d35884b5a84c24957bbbfa1a7 (127.0.97.3:37177)
I20260619 01:11:25.481657 18109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:48093
I20260619 01:11:25.491601 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:11:25.501634 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.495599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.0.97.62:46829)
0619 01:11:25.495807 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:25.495812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:25.496236 (+ 424us) server_negotiation.cc:407] Connection header received
0619 01:11:25.496381 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:25.496384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:25.496482 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:25.496561 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:25.497028 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.497858 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.499411 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.499823 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.499923 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.500958 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:25.500963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:25.500966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:25.500986 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:25.501019 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:25.501031 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:25.501101 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:25.501220 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 01:11:25.501362 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:11:25.505649 18109 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:11:25.512470 18109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:11:25.541934 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.525663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.0.97.3:37177)
0619 01:11:25.533608 (+ 7945us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:25.533616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:25.533643 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:11:25.533719 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:25.533724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:25.533911 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:25.534080 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:25.534932 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.536231 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.539110 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.540606 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.540756 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.541058 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:25.541343 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 01:11:25.541533 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7771,"thread_start_us":7617,"threads_started":1}
I20260619 01:11:25.542169 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.527206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.0.97.1:34425)
0619 01:11:25.532480 (+ 5274us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:25.532492 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:25.533492 (+ 1000us) server_negotiation.cc:407] Connection header received
0619 01:11:25.533605 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:25.533613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:25.533860 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:25.534068 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:25.535092 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.536816 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.539517 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.540922 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.541065 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.541375 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:25.541633 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 01:11:25.541793 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3812,"thread_start_us":3681,"threads_started":1}
I20260619 01:11:25.542196 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.526451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34425 (local address 127.0.0.1:57434)
0619 01:11:25.533180 (+ 6729us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:25.533201 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:25.533352 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:25.534353 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:25.534361 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:25.534412 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:25.534894 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.534904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.537025 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.537031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.539312 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.539327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.541199 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.541205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.541443 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.541476 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 01:11:25.541578 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 01:11:25.541784 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6584,"mutex_wait_us":30,"thread_start_us":5589,"threads_started":1}
I20260619 01:11:25.542749 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.534014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41827 (local address 127.0.0.1:57950)
0619 01:11:25.534742 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:25.534762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:25.534897 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:25.535638 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:25.535644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:25.535700 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:25.536042 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.536053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.538243 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.538249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.540031 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.540042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.541954 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.541962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.542159 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.542191 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:11:25.542279 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 01:11:25.542426 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":405,"threads_started":1}
I20260619 01:11:25.543208 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.525564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37177 (local address 127.0.0.1:54222)
0619 01:11:25.531902 (+ 6338us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:25.531975 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:25.532538 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:25.534121 (+ 1583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:25.534132 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:25.534174 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:25.534682 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.534700 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.536379 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.536386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.538457 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.538476 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.540813 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.540820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.541033 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.541089 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 01:11:25.542681 (+ 1592us) client_negotiation.cc:240] Negotiation successful
0619 01:11:25.542840 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5983,"mutex_wait_us":1,"thread_start_us":7342,"threads_started":1}
I20260619 01:11:25.543322 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.533388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.0.97.2:41827)
0619 01:11:25.533973 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:25.533981 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:25.535032 (+ 1051us) server_negotiation.cc:407] Connection header received
0619 01:11:25.535133 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:25.535140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:25.535397 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:25.536211 (+ 814us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:25.536431 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.538073 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.540218 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.541475 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.541594 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.541820 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:25.542772 (+ 952us) server_negotiation.cc:299] Negotiation successful
0619 01:11:25.542944 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":297,"threads_started":1}
I20260619 01:11:25.546705 18230 tablet_service.cc:1467] Processing CreateTablet for tablet 7052100f3d7e496298add3ac6ef0517e (DEFAULT_TABLE table=test-table [id=a8f66c48d6684781858e6a3aca0f2838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:25.546887 18361 tablet_service.cc:1467] Processing CreateTablet for tablet 7052100f3d7e496298add3ac6ef0517e (DEFAULT_TABLE table=test-table [id=a8f66c48d6684781858e6a3aca0f2838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:25.546886 18491 tablet_service.cc:1467] Processing CreateTablet for tablet 7052100f3d7e496298add3ac6ef0517e (DEFAULT_TABLE table=test-table [id=a8f66c48d6684781858e6a3aca0f2838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:25.548180 18230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7052100f3d7e496298add3ac6ef0517e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:25.548275 18491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7052100f3d7e496298add3ac6ef0517e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:25.548364 18361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7052100f3d7e496298add3ac6ef0517e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:25.548758 18295 heartbeater.cc:503] Master 127.0.97.62:46829 was elected leader, sending a full tablet report...
I20260619 01:11:25.559908 18574 tablet_bootstrap.cc:492] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: Bootstrap starting.
I20260619 01:11:25.563517 18575 tablet_bootstrap.cc:492] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: Bootstrap starting.
I20260619 01:11:25.563840 18574 tablet_bootstrap.cc:654] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:25.564318 18576 tablet_bootstrap.cc:492] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: Bootstrap starting.
I20260619 01:11:25.565333 18574 log.cc:824] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:25.567930 18575 tablet_bootstrap.cc:654] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:25.568667 18576 tablet_bootstrap.cc:654] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:25.569054 18575 log.cc:824] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:25.570184 18576 log.cc:824] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:25.572801 18574 tablet_bootstrap.cc:492] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: No bootstrap required, opened a new log
I20260619 01:11:25.573064 18575 tablet_bootstrap.cc:492] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: No bootstrap required, opened a new log
I20260619 01:11:25.573169 18574 ts_tablet_manager.cc:1397] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260619 01:11:25.573566 18575 ts_tablet_manager.cc:1397] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 01:11:25.580134 18576 tablet_bootstrap.cc:492] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: No bootstrap required, opened a new log
I20260619 01:11:25.580466 18576 ts_tablet_manager.cc:1397] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260619 01:11:25.581477 18574 raft_consensus.cc:348] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } } peers { permanent_uuid: "55fc89253eaf431bbff17fb36231bf85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34425 } } peers { permanent_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41827 } }
I20260619 01:11:25.582036 18574 raft_consensus.cc:374] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:25.582161 18574 raft_consensus.cc:729] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc89253eaf431bbff17fb36231bf85, State: Initialized, Role: FOLLOWER
I20260619 01:11:25.582878 18574 consensus_queue.cc:260] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } } peers { permanent_uuid: "55fc89253eaf431bbff17fb36231bf85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34425 } } peers { permanent_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41827 } }
I20260619 01:11:25.584693 18574 ts_tablet_manager.cc:1428] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 01:11:25.585824 18575 raft_consensus.cc:348] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } } peers { permanent_uuid: "55fc89253eaf431bbff17fb36231bf85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34425 } } peers { permanent_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41827 } }
I20260619 01:11:25.586359 18575 raft_consensus.cc:374] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:25.586465 18575 raft_consensus.cc:729] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6114ad9b9d4a59bdb2da68165e5f18, State: Initialized, Role: FOLLOWER
I20260619 01:11:25.587132 18575 consensus_queue.cc:260] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } } peers { permanent_uuid: "55fc89253eaf431bbff17fb36231bf85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34425 } } peers { permanent_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41827 } }
I20260619 01:11:25.588289 18426 heartbeater.cc:503] Master 127.0.97.62:46829 was elected leader, sending a full tablet report...
I20260619 01:11:25.589044 18575 ts_tablet_manager.cc:1428] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260619 01:11:25.590672 18576 raft_consensus.cc:348] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } } peers { permanent_uuid: "55fc89253eaf431bbff17fb36231bf85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34425 } } peers { permanent_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41827 } }
I20260619 01:11:25.591153 18576 raft_consensus.cc:374] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:25.591260 18576 raft_consensus.cc:729] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e9d82d35884b5a84c24957bbbfa1a7, State: Initialized, Role: FOLLOWER
I20260619 01:11:25.591804 18576 consensus_queue.cc:260] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } } peers { permanent_uuid: "55fc89253eaf431bbff17fb36231bf85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34425 } } peers { permanent_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41827 } }
I20260619 01:11:25.593240 18556 heartbeater.cc:503] Master 127.0.97.62:46829 was elected leader, sending a full tablet report...
I20260619 01:11:25.594002 18576 ts_tablet_manager.cc:1428] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260619 01:11:25.695369 18581 raft_consensus.cc:482] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:11:25.695706 18581 raft_consensus.cc:504] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } } peers { permanent_uuid: "55fc89253eaf431bbff17fb36231bf85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34425 } } peers { permanent_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41827 } }
I20260619 01:11:25.697655 18581 leader_election.cc:288] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e9d82d35884b5a84c24957bbbfa1a7 (127.0.97.3:37177), 55fc89253eaf431bbff17fb36231bf85 (127.0.97.1:34425)
I20260619 01:11:25.705391 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.698353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56397 (local address 127.0.97.1:34425)
0619 01:11:25.698576 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:25.698584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:25.699214 (+ 630us) server_negotiation.cc:407] Connection header received
0619 01:11:25.699465 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:25.699474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:25.699602 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:25.699714 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:25.700660 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.701435 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.703514 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.704310 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.704493 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.704619 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:25.704953 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 01:11:25.705091 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 01:11:25.705391 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.697974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33057 (local address 127.0.97.3:37177)
0619 01:11:25.698211 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:25.698222 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:25.699549 (+ 1327us) server_negotiation.cc:407] Connection header received
0619 01:11:25.699751 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:25.699756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:25.699834 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:25.699921 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:25.700550 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.701434 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.703528 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.704348 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.704522 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.704629 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:25.704947 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 01:11:25.705087 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 01:11:25.705652 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.697945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37177 (local address 127.0.97.2:33057)
0619 01:11:25.699441 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:25.699455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:25.699601 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:25.699995 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:25.699999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:25.700022 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:25.700372 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.700380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.701563 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.701570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.703301 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.703314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.704568 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.704575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.704783 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.704809 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 01:11:25.704922 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 01:11:25.705097 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"mutex_wait_us":132,"thread_start_us":629,"threads_started":1}
I20260619 01:11:25.705803 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.697908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34425 (local address 127.0.97.2:56397)
0619 01:11:25.699068 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:25.699085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:25.699307 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:25.700044 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:25.700048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:25.700071 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:25.700524 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.700534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.701572 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.701580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.703295 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:25.703308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.704565 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.704573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:25.704777 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.704804 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 01:11:25.704925 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 01:11:25.705090 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":110,"thread_start_us":1266,"threads_started":1}
I20260619 01:11:25.706760 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7052100f3d7e496298add3ac6ef0517e" candidate_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" is_pre_election: true
I20260619 01:11:25.706842 18250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7052100f3d7e496298add3ac6ef0517e" candidate_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc89253eaf431bbff17fb36231bf85" is_pre_election: true
I20260619 01:11:25.707329 18511 raft_consensus.cc:2454] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6114ad9b9d4a59bdb2da68165e5f18 in term 0.
I20260619 01:11:25.707342 18250 raft_consensus.cc:2454] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6114ad9b9d4a59bdb2da68165e5f18 in term 0.
I20260619 01:11:25.708088 18317 leader_election.cc:302] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55fc89253eaf431bbff17fb36231bf85, 85e9d82d35884b5a84c24957bbbfa1a7, af6114ad9b9d4a59bdb2da68165e5f18; no voters:
I20260619 01:11:25.708544 18581 raft_consensus.cc:2781] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:11:25.708706 18581 raft_consensus.cc:482] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:11:25.708813 18581 raft_consensus.cc:3037] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:25.711344 18581 raft_consensus.cc:504] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } } peers { permanent_uuid: "55fc89253eaf431bbff17fb36231bf85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34425 } } peers { permanent_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41827 } }
I20260619 01:11:25.711984 18581 leader_election.cc:288] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [CANDIDATE]: Term 1 election: Requested vote from peers 85e9d82d35884b5a84c24957bbbfa1a7 (127.0.97.3:37177), 55fc89253eaf431bbff17fb36231bf85 (127.0.97.1:34425)
I20260619 01:11:25.712569 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7052100f3d7e496298add3ac6ef0517e" candidate_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e9d82d35884b5a84c24957bbbfa1a7"
I20260619 01:11:25.712582 18250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7052100f3d7e496298add3ac6ef0517e" candidate_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc89253eaf431bbff17fb36231bf85"
I20260619 01:11:25.712808 18511 raft_consensus.cc:3037] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:25.712920 18250 raft_consensus.cc:3037] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85 [term 0 FOLLOWER]: Advancing to term 1
W20260619 01:11:25.714437 18557 tablet.cc:2369] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:11:25.715436 18250 raft_consensus.cc:2454] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6114ad9b9d4a59bdb2da68165e5f18 in term 1.
I20260619 01:11:25.715883 18511 raft_consensus.cc:2454] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6114ad9b9d4a59bdb2da68165e5f18 in term 1.
I20260619 01:11:25.716002 18317 leader_election.cc:302] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55fc89253eaf431bbff17fb36231bf85, af6114ad9b9d4a59bdb2da68165e5f18; no voters:
I20260619 01:11:25.716356 18581 raft_consensus.cc:2781] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:25.716804 18581 raft_consensus.cc:686] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 1 LEADER]: Becoming Leader. State: Replica: af6114ad9b9d4a59bdb2da68165e5f18, State: Running, Role: LEADER
I20260619 01:11:25.717361 18581 consensus_queue.cc:237] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } } peers { permanent_uuid: "55fc89253eaf431bbff17fb36231bf85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34425 } } peers { permanent_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41827 } }
I20260619 01:11:25.722414 18108 catalog_manager.cc:5495] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 reported cstate change: term changed from 0 to 1, leader changed from <none> to af6114ad9b9d4a59bdb2da68165e5f18 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55fc89253eaf431bbff17fb36231bf85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af6114ad9b9d4a59bdb2da68165e5f18" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41827 } health_report { overall_health: HEALTHY } } }
W20260619 01:11:25.737190 18427 tablet.cc:2369] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:11:25.776335 18296 tablet.cc:2369] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:11:25.782809 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.777443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.0.97.2:41827)
0619 01:11:25.777638 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:25.777642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:25.777659 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:11:25.777746 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:25.777750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:25.777894 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:25.778054 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:11:25.778441 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.779128 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.780561 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.780981 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.781135 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.781703 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:11:25.782231 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:11:25.782394 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:25.782447 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 01:11:25.782529 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:11:25.791721 18511 raft_consensus.cc:1261] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7 [term 1 FOLLOWER]: Refusing update from remote peer af6114ad9b9d4a59bdb2da68165e5f18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:25.791721 18250 raft_consensus.cc:1261] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85 [term 1 FOLLOWER]: Refusing update from remote peer af6114ad9b9d4a59bdb2da68165e5f18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:25.792968 18585 consensus_queue.cc:1035] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e9d82d35884b5a84c24957bbbfa1a7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:25.793398 18581 consensus_queue.cc:1035] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc89253eaf431bbff17fb36231bf85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:25.811626 18589 mvcc.cc:205] Tried to move back new op lower bound from 7298381765790339072 to 7298381765502423040. Current Snapshot: MvccSnapshot[applied={T|T < 7298381765790339072}]
I20260619 01:11:25.834306 18108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:25.834614 18108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:25.838243 18108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4eb5e20ff2c4c0e86d9fa2f91ffcde7: Sending DeleteTablet for 3 replicas of tablet 7052100f3d7e496298add3ac6ef0517e
I20260619 01:11:25.839444 18491 tablet_service.cc:1514] Processing DeleteTablet for tablet 7052100f3d7e496298add3ac6ef0517e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:25 UTC) from {username='slave'} at 127.0.0.1:54222
I20260619 01:11:25.839599 18230 tablet_service.cc:1514] Processing DeleteTablet for tablet 7052100f3d7e496298add3ac6ef0517e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:25 UTC) from {username='slave'} at 127.0.0.1:57434
I20260619 01:11:25.840565 18599 tablet_replica.cc:330] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: stopping tablet replica
I20260619 01:11:25.840937 18361 tablet_service.cc:1514] Processing DeleteTablet for tablet 7052100f3d7e496298add3ac6ef0517e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:25 UTC) from {username='slave'} at 127.0.0.1:57950
I20260619 01:11:25.841025 18599 raft_consensus.cc:2229] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:25.841045 18598 tablet_replica.cc:330] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: stopping tablet replica
I20260619 01:11:25.841562 18599 raft_consensus.cc:2258] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:25.841548 18598 raft_consensus.cc:2229] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:25.841694 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 18168
I20260619 01:11:25.842351 18598 raft_consensus.cc:2258] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:25.845597 18599 ts_tablet_manager.cc:1905] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:25.850783 18598 ts_tablet_manager.cc:1905] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:25.852600 18599 ts_tablet_manager.cc:1918] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:25.852768 18599 log.cc:1196] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/7052100f3d7e496298add3ac6ef0517e
I20260619 01:11:25.853837 18600 tablet_replica.cc:330] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: stopping tablet replica
I20260619 01:11:25.854279 18600 raft_consensus.cc:2229] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:11:25.854946 18600 raft_consensus.cc:2258] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:25.857141 18600 ts_tablet_manager.cc:1905] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:25.860700 18598 ts_tablet_manager.cc:1918] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:25.860942 18598 log.cc:1196] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/7052100f3d7e496298add3ac6ef0517e
I20260619 01:11:25.861577 18598 ts_tablet_manager.cc:1939] T 7052100f3d7e496298add3ac6ef0517e P 85e9d82d35884b5a84c24957bbbfa1a7: Deleting consensus metadata
I20260619 01:11:25.861826 18599 ts_tablet_manager.cc:1939] T 7052100f3d7e496298add3ac6ef0517e P 55fc89253eaf431bbff17fb36231bf85: Deleting consensus metadata
I20260619 01:11:25.863696 18094 catalog_manager.cc:4841] TS 85e9d82d35884b5a84c24957bbbfa1a7 (127.0.97.3:37177): tablet 7052100f3d7e496298add3ac6ef0517e (table test-table [id=a8f66c48d6684781858e6a3aca0f2838]) successfully deleted
I20260619 01:11:25.863705 18096 catalog_manager.cc:4841] TS 55fc89253eaf431bbff17fb36231bf85 (127.0.97.1:34425): tablet 7052100f3d7e496298add3ac6ef0517e (table test-table [id=a8f66c48d6684781858e6a3aca0f2838]) successfully deleted
I20260619 01:11:25.864483 18600 ts_tablet_manager.cc:1918] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:25.864667 18600 log.cc:1196] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/7052100f3d7e496298add3ac6ef0517e
I20260619 01:11:25.865157 18600 ts_tablet_manager.cc:1939] T 7052100f3d7e496298add3ac6ef0517e P af6114ad9b9d4a59bdb2da68165e5f18: Deleting consensus metadata
I20260619 01:11:25.866180 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.842744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57444 (local address 127.0.97.1:34425)
0619 01:11:25.842950 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:25.842958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:25.855782 (+ 12824us) server_negotiation.cc:407] Connection header received
0619 01:11:25.856084 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:25.856090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:25.856206 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:25.856339 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:25.856992 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.858192 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.860201 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.860832 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.860930 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.864409 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:25.864421 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:25.864439 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:25.865222 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:25.865329 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:25.865355 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:25.865441 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:25.865708 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 01:11:25.865828 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 01:11:25.866888 18096 catalog_manager.cc:4841] TS af6114ad9b9d4a59bdb2da68165e5f18 (127.0.97.2:41827): tablet 7052100f3d7e496298add3ac6ef0517e (table test-table [id=a8f66c48d6684781858e6a3aca0f2838]) successfully deleted
I20260619 01:11:25.866917 18290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:25.966720 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 18168
I20260619 01:11:25.991431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 18299
I20260619 01:11:26.000941 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:25.992404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.0.97.2:41827)
0619 01:11:25.992598 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:25.992604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:25.992630 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:11:25.992822 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:25.992827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:25.992943 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:25.993064 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:25.993534 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.994796 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.996742 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:25.997671 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:25.997804 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:25.999092 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:25.999104 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:25.999123 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:26.000054 (+ 931us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:26.000127 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:26.000157 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:26.000281 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:26.000440 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 01:11:26.000555 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:11:26.001816 18421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:26.117343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 18299
I20260619 01:11:26.136380 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 18429
I20260619 01:11:26.143328 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:26.137246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.0.97.3:37177)
0619 01:11:26.137447 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:26.137451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:26.137470 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:11:26.137615 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:26.137619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:26.137712 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:26.137803 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:26.138204 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:26.138914 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:26.140535 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:26.140926 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:26.141029 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:26.142093 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:26.142103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:26.142119 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:26.142723 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:26.142773 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:26.142797 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:26.142890 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:26.142981 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 01:11:26.143072 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 01:11:26.144034 18551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:26.232581 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 18429
I20260619 01:11:26.249653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 18078
I20260619 01:11:26.250766 18139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:26.361097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 18078
2026-06-19T01:11:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2946 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:26 dist-test-slave-5p8w krb5kdc[18608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:26 dist-test-slave-5p8w krb5kdc[18608](info): set up 2 sockets
Jun 19 01:11:26 dist-test-slave-5p8w krb5kdc[18608](info): commencing operation
krb5kdc: starting...
W20260619 01:11:27.489570 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:27 dist-test-slave-5p8w krb5kdc[18608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:11:27.744375 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37159
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:28.074210 18621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:28.074580 18621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:28.074751 18621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:28.082170 18621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:28.082350 18621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:28.082464 18621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:28.082571 18621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:28.082659 18621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:28.092769 18621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:28.094499 18621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:28.095881 18621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:28.107671 18629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:28.107754 18626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:28.107741 18627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:28.108789 18621 server_base.cc:1034] running on GCE node
I20260619 01:11:28.109688 18621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:28.111171 18621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:28.112399 18621 hybrid_clock.cc:648] HybridClock initialized: now 1781831488112332 us; error 61 us; skew 500 ppm
I20260619 01:11:28.112742 18621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:28 dist-test-slave-5p8w krb5kdc[18608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:28.116004 18621 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:11:28.117606 18621 webserver.cc:460] Webserver started at http://127.0.97.62:34005/ using document root <none> and password file <none>
I20260619 01:11:28.118268 18621 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:28.118366 18621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:28.118624 18621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:28.121665 18621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "2b88f98d0a87444dab8cc8007ea81c7a"
format_stamp: "Formatted at 2026-06-19 01:11:28 on dist-test-slave-5p8w"
server_key: "98b42c055666bf04af6e993a63e857c1"
server_key_iv: "a68162208329cbe3d73f3f6edca6f65a"
server_key_version: "encryptionkey@0"
I20260619 01:11:28.122381 18621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "2b88f98d0a87444dab8cc8007ea81c7a"
format_stamp: "Formatted at 2026-06-19 01:11:28 on dist-test-slave-5p8w"
server_key: "98b42c055666bf04af6e993a63e857c1"
server_key_iv: "a68162208329cbe3d73f3f6edca6f65a"
server_key_version: "encryptionkey@0"
I20260619 01:11:28.127638 18621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260619 01:11:28.131103 18636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:28.132432 18621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 01:11:28.132629 18621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "2b88f98d0a87444dab8cc8007ea81c7a"
format_stamp: "Formatted at 2026-06-19 01:11:28 on dist-test-slave-5p8w"
server_key: "98b42c055666bf04af6e993a63e857c1"
server_key_iv: "a68162208329cbe3d73f3f6edca6f65a"
server_key_version: "encryptionkey@0"
I20260619 01:11:28.132839 18621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:28.157662 18621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:28.162305 18621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:28.162626 18621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:28.180967 18621 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37159
I20260619 01:11:28.180998 18687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37159 every 8 connection(s)
I20260619 01:11:28.182420 18621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:28.186316 18688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:28.188638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 18621
I20260619 01:11:28.188818 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:28.189376 388 external_mini_cluster.cc:1428] Setting key b29e062f7c4c952e8544b31049c27deb
I20260619 01:11:28.199333 18688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a: Bootstrap starting.
I20260619 01:11:28.203485 18688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:28.204878 18688 log.cc:824] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a: Log is configured to *not* fsync() on all Append() calls
Jun 19 01:11:28 dist-test-slave-5p8w krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:28.208045 18688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a: No bootstrap required, opened a new log
I20260619 01:11:28.214447 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:28.191297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.0.97.62:37159)
0619 01:11:28.192179 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:28.192191 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:28.192236 (+ 45us) server_negotiation.cc:407] Connection header received
0619 01:11:28.194022 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:28.194050 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:28.194460 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:28.194791 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:28.196698 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:28.198229 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:28.200318 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:28.201063 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:28.201213 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:28.206650 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:28.206659 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:28.206675 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:28.206715 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:28.209996 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:28.210754 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:28.210762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:28.210768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:28.210963 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:28.211383 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:28.211397 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:28.211400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:28.211970 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:28.212362 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:28.212801 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 01:11:28.213270 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":394,"threads_started":1}
I20260619 01:11:28.216239 18688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88f98d0a87444dab8cc8007ea81c7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } }
I20260619 01:11:28.216723 18688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:28.216832 18688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b88f98d0a87444dab8cc8007ea81c7a, State: Initialized, Role: FOLLOWER
I20260619 01:11:28.217542 18688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88f98d0a87444dab8cc8007ea81c7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } }
I20260619 01:11:28.217824 18688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:28.217934 18688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:28.218058 18688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:28.220168 18688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88f98d0a87444dab8cc8007ea81c7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } }
I20260619 01:11:28.220616 18688 leader_election.cc:302] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b88f98d0a87444dab8cc8007ea81c7a; no voters:
I20260619 01:11:28.221204 18688 leader_election.cc:288] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:28.221371 18693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:28.223023 18693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [term 1 LEADER]: Becoming Leader. State: Replica: 2b88f98d0a87444dab8cc8007ea81c7a, State: Running, Role: LEADER
I20260619 01:11:28.223497 18688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:28.223712 18693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88f98d0a87444dab8cc8007ea81c7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } }
I20260619 01:11:28.226600 18694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b88f98d0a87444dab8cc8007ea81c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88f98d0a87444dab8cc8007ea81c7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } } }
I20260619 01:11:28.227262 18694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:28.226635 18695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b88f98d0a87444dab8cc8007ea81c7a. Latest consensus state: current_term: 1 leader_uuid: "2b88f98d0a87444dab8cc8007ea81c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88f98d0a87444dab8cc8007ea81c7a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37159 } } }
I20260619 01:11:28.227748 18695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:28.235157 18702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:28.239923 18702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:28.249634 18702 catalog_manager.cc:1347] Generated new cluster ID: a3385cdedaf845e58601d90f6eea389f
I20260619 01:11:28.249799 18702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:28.266501 18702 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:28.267977 18702 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:28.274279 18702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a: Generated new TSK 0
I20260619 01:11:28.275209 18702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:11:28.344230 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37159
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:28.697371 18714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:28.697701 18714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:28.697808 18714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:28.705473 18714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:28.705608 18714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:28.705742 18714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:11:28.716629 18714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:28.718081 18714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:28.719450 18714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:28.733773 18719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:28.733827 18720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:28.733762 18722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:28.735435 18714 server_base.cc:1034] running on GCE node
I20260619 01:11:28.736133 18714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:28.737273 18714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:28.738623 18714 hybrid_clock.cc:648] HybridClock initialized: now 1781831488738561 us; error 54 us; skew 500 ppm
I20260619 01:11:28.738987 18714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:28 dist-test-slave-5p8w krb5kdc[18608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:28.742455 18714 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:11:28.744171 18714 webserver.cc:460] Webserver started at http://127.0.97.1:42169/ using document root <none> and password file <none>
I20260619 01:11:28.744923 18714 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:28.745028 18714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:28.745386 18714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:28.748667 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "6609b2e6290543cfbc36e5da255f6aed"
format_stamp: "Formatted at 2026-06-19 01:11:28 on dist-test-slave-5p8w"
server_key: "2a115ea1c5b52365b2448d4445241254"
server_key_iv: "79a346024cb40580973237036b4828d6"
server_key_version: "encryptionkey@0"
I20260619 01:11:28.749590 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6609b2e6290543cfbc36e5da255f6aed"
format_stamp: "Formatted at 2026-06-19 01:11:28 on dist-test-slave-5p8w"
server_key: "2a115ea1c5b52365b2448d4445241254"
server_key_iv: "79a346024cb40580973237036b4828d6"
server_key_version: "encryptionkey@0"
I20260619 01:11:28.755234 18714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:11:28.759092 18729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:28.760699 18714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 01:11:28.760848 18714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "6609b2e6290543cfbc36e5da255f6aed"
format_stamp: "Formatted at 2026-06-19 01:11:28 on dist-test-slave-5p8w"
server_key: "2a115ea1c5b52365b2448d4445241254"
server_key_iv: "79a346024cb40580973237036b4828d6"
server_key_version: "encryptionkey@0"
I20260619 01:11:28.761039 18714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:28.778092 18714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:28.782634 18714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:28.782903 18714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:28.783921 18714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:28.785542 18714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:28.785619 18714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:28.785709 18714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:28.785755 18714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:28.820679 18714 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34747
I20260619 01:11:28.820725 18841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34747 every 8 connection(s)
I20260619 01:11:28.822116 18714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:11:28.830322 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 18714
I20260619 01:11:28.830524 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:11:28.830976 388 external_mini_cluster.cc:1428] Setting key 003b748bef9f094f986ea76e6f0e387e
Jun 19 01:11:28 dist-test-slave-5p8w krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:28.844810 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:28.824837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34613 (local address 127.0.97.62:37159)
0619 01:11:28.825286 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:28.825333 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:28.826280 (+ 947us) server_negotiation.cc:407] Connection header received
0619 01:11:28.828227 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:28.828233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:28.828342 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:28.828469 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:28.830252 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:28.831148 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:28.833250 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:28.833762 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:28.833899 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:28.837764 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:28.837770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:28.837774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:28.837809 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:28.841130 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:28.842277 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:28.842281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:28.842283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:28.842413 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:28.842963 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:28.842966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:28.842968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:28.843266 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:28.843419 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:28.844357 (+ 938us) server_negotiation.cc:299] Negotiation successful
0619 01:11:28.844557 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":243,"threads_started":1}
I20260619 01:11:28.846482 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:28.825350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37159 (local address 127.0.97.1:34613)
0619 01:11:28.826095 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:28.826139 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:28.827917 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:28.828727 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:28.828735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:28.829359 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:28.830037 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:28.830065 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:28.831325 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:28.831330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:28.833075 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:28.833116 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:28.833999 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:28.834010 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:28.834285 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:28.835036 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:11:28.835061 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:28.837519 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:28.841355 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:28.841366 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:28.841387 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:28.842131 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:28.842539 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:28.842547 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:28.842553 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:28.842845 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:28.843408 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:28.843420 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:28.843928 (+ 508us) client_negotiation.cc:769] Sending connection context
0619 01:11:28.844260 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 01:11:28.844903 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":304,"threads_started":1}
I20260619 01:11:28.848377 18842 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37159
I20260619 01:11:28.848738 18842 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:28.849746 18842 heartbeater.cc:511] Master 127.0.97.62:37159 requested a full tablet report, sending...
I20260619 01:11:28.852449 18653 ts_manager.cc:194] Registered new tserver with Master: 6609b2e6290543cfbc36e5da255f6aed (127.0.97.1:34747)
I20260619 01:11:28.854792 18653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34613
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:11:28.892253 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37159
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:29.229807 18848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:29.230203 18848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:29.230388 18848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:29.238221 18848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:29.238386 18848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:29.238570 18848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:11:29.249048 18848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:29.250592 18848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:29.252128 18848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:29.265898 18853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:29.265892 18854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:29.265923 18856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:29.267001 18848 server_base.cc:1034] running on GCE node
I20260619 01:11:29.267748 18848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:29.268786 18848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:29.270062 18848 hybrid_clock.cc:648] HybridClock initialized: now 1781831489270011 us; error 30 us; skew 500 ppm
I20260619 01:11:29.270507 18848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:29 dist-test-slave-5p8w krb5kdc[18608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:29.281683 18848 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:11:29.283438 18848 webserver.cc:460] Webserver started at http://127.0.97.2:38683/ using document root <none> and password file <none>
I20260619 01:11:29.284221 18848 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:29.284374 18848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:29.284719 18848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:29.287963 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "50a60bb832e3487c91a7e2c6d913f15d"
format_stamp: "Formatted at 2026-06-19 01:11:29 on dist-test-slave-5p8w"
server_key: "84234398ac46aafc888f4920e8a79e55"
server_key_iv: "086b005afd4b9499a75ba126526bea0b"
server_key_version: "encryptionkey@0"
I20260619 01:11:29.288832 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "50a60bb832e3487c91a7e2c6d913f15d"
format_stamp: "Formatted at 2026-06-19 01:11:29 on dist-test-slave-5p8w"
server_key: "84234398ac46aafc888f4920e8a79e55"
server_key_iv: "086b005afd4b9499a75ba126526bea0b"
server_key_version: "encryptionkey@0"
I20260619 01:11:29.294628 18848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 01:11:29.298503 18863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:29.300053 18848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:11:29.300253 18848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "50a60bb832e3487c91a7e2c6d913f15d"
format_stamp: "Formatted at 2026-06-19 01:11:29 on dist-test-slave-5p8w"
server_key: "84234398ac46aafc888f4920e8a79e55"
server_key_iv: "086b005afd4b9499a75ba126526bea0b"
server_key_version: "encryptionkey@0"
I20260619 01:11:29.300494 18848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:29.327543 18848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:29.332091 18848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:29.332398 18848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:29.333506 18848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:29.335000 18848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:29.335109 18848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:29.335260 18848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:29.335335 18848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:29.370687 18848 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34009
I20260619 01:11:29.370718 18975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34009 every 8 connection(s)
I20260619 01:11:29.372522 18848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:11:29.381181 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 18848
I20260619 01:11:29.381383 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:11:29.381855 388 external_mini_cluster.cc:1428] Setting key ae0969b2866c80d6a2a5630ac28db47f
Jun 19 01:11:29 dist-test-slave-5p8w krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:29.395964 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:29.375199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56007 (local address 127.0.97.62:37159)
0619 01:11:29.375678 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:29.375684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:29.376763 (+ 1079us) server_negotiation.cc:407] Connection header received
0619 01:11:29.378709 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:29.378714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:29.378796 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:29.378895 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:29.380428 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:29.381451 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:29.383212 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:29.383917 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:29.384048 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:29.388017 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:29.388023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:29.388026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:29.388063 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:29.392079 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:29.393257 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:29.393262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:29.393264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:29.393475 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:29.394033 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:29.394037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:29.394038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:29.394312 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:29.394427 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:29.395464 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0619 01:11:29.395656 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260619 01:11:29.397653 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:29.375845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37159 (local address 127.0.97.2:56007)
0619 01:11:29.376586 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:29.376631 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:29.378430 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:29.379128 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:29.379136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:29.379695 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:29.380269 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:29.380285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:29.381587 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:29.381591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:29.383075 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:29.383087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:29.384155 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:29.384162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:29.384396 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:29.385101 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:11:29.385123 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:29.387716 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:29.392338 (+ 4622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:29.392353 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:29.392377 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:29.393086 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:29.393595 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:29.393601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:29.393605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:29.393913 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:29.394466 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:29.394477 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:29.395011 (+ 534us) client_negotiation.cc:769] Sending connection context
0619 01:11:29.395347 (+ 336us) client_negotiation.cc:240] Negotiation successful
0619 01:11:29.395992 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":280,"threads_started":1}
I20260619 01:11:29.399324 18976 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37159
I20260619 01:11:29.399662 18976 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:29.400457 18976 heartbeater.cc:511] Master 127.0.97.62:37159 requested a full tablet report, sending...
I20260619 01:11:29.402145 18653 ts_manager.cc:194] Registered new tserver with Master: 50a60bb832e3487c91a7e2c6d913f15d (127.0.97.2:34009)
I20260619 01:11:29.403007 18653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56007
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:11:29.442972 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37159
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:29.778833 18982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:29.779203 18982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:29.779386 18982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:29.787375 18982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:29.787544 18982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:29.787719 18982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:11:29.798327 18982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:29.799713 18982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:29.801138 18982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:29.814884 18987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:29.814909 18990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:29.814940 18988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:29.815922 18982 server_base.cc:1034] running on GCE node
I20260619 01:11:29.816553 18982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:29.817602 18982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:29.818848 18982 hybrid_clock.cc:648] HybridClock initialized: now 1781831489818791 us; error 49 us; skew 500 ppm
I20260619 01:11:29.819199 18982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:29 dist-test-slave-5p8w krb5kdc[18608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:29.822724 18982 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:11:29.824455 18982 webserver.cc:460] Webserver started at http://127.0.97.3:37575/ using document root <none> and password file <none>
I20260619 01:11:29.825428 18982 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:29.825579 18982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:29.825919 18982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:29.828994 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "9aebe42bba7a4f8ab82037a87258d220"
format_stamp: "Formatted at 2026-06-19 01:11:29 on dist-test-slave-5p8w"
server_key: "19a1e3b37583ff8222a7084a36aebcdc"
server_key_iv: "616574e19a5bddea5521ce4309627832"
server_key_version: "encryptionkey@0"
I20260619 01:11:29.829890 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9aebe42bba7a4f8ab82037a87258d220"
format_stamp: "Formatted at 2026-06-19 01:11:29 on dist-test-slave-5p8w"
server_key: "19a1e3b37583ff8222a7084a36aebcdc"
server_key_iv: "616574e19a5bddea5521ce4309627832"
server_key_version: "encryptionkey@0"
I20260619 01:11:29.835536 18982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:11:29.839663 18997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:29.841144 18982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:11:29.841380 18982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "9aebe42bba7a4f8ab82037a87258d220"
format_stamp: "Formatted at 2026-06-19 01:11:29 on dist-test-slave-5p8w"
server_key: "19a1e3b37583ff8222a7084a36aebcdc"
server_key_iv: "616574e19a5bddea5521ce4309627832"
server_key_version: "encryptionkey@0"
I20260619 01:11:29.841614 18982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:29.859221 18842 heartbeater.cc:503] Master 127.0.97.62:37159 was elected leader, sending a full tablet report...
I20260619 01:11:29.866739 18982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:29.871094 18982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:29.871407 18982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:29.872442 18982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:29.873849 18982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:29.873965 18982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:29.874120 18982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:29.874192 18982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:29.908054 18982 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40129
I20260619 01:11:29.908095 19109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40129 every 8 connection(s)
I20260619 01:11:29.909468 18982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:11:29.918076 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 18982
I20260619 01:11:29.918253 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:11:29.918699 388 external_mini_cluster.cc:1428] Setting key 338bc9995fa9d5a8088d22601c8496f6
Jun 19 01:11:29 dist-test-slave-5p8w krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:29.930682 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:29.912062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43147 (local address 127.0.97.62:37159)
0619 01:11:29.912566 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:29.912581 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:29.913654 (+ 1073us) server_negotiation.cc:407] Connection header received
0619 01:11:29.915550 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:29.915555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:29.915661 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:29.915794 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:29.917819 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:29.918713 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:29.920678 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:29.921107 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:29.921257 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:29.924985 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:29.924990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:29.924994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:29.925028 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:29.927813 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:29.928540 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:29.928544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:29.928547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:29.928661 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:29.929045 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:29.929047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:29.929049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:29.929382 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:29.929530 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:29.930207 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 01:11:29.930388 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":220,"threads_started":1}
I20260619 01:11:29.931676 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:29.912692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37159 (local address 127.0.97.3:43147)
0619 01:11:29.913484 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:29.913530 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:29.915233 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:29.916102 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:29.916114 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:29.916929 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:29.917619 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:29.917638 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:29.918883 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:29.918889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:29.920524 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:29.920540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:29.921468 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:29.921476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:29.921694 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:29.922403 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:11:29.922428 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:29.924761 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:29.927958 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:29.927965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:29.927979 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:29.928424 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:29.928751 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:29.928754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:29.928756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:29.928938 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:29.929492 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:29.929498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:29.929839 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 01:11:29.930083 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 01:11:29.930501 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":294,"threads_started":1}
I20260619 01:11:29.933383 19110 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37159
I20260619 01:11:29.933712 19110 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:29.934592 19110 heartbeater.cc:511] Master 127.0.97.62:37159 requested a full tablet report, sending...
I20260619 01:11:29.936112 18652 ts_manager.cc:194] Registered new tserver with Master: 9aebe42bba7a4f8ab82037a87258d220 (127.0.97.3:40129)
I20260619 01:11:29.937099 18652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43147
I20260619 01:11:29.946712 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:11:29.964468 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:29.950762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.0.97.62:37159)
0619 01:11:29.950981 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:29.950989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:29.951343 (+ 354us) server_negotiation.cc:407] Connection header received
0619 01:11:29.951549 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:29.951555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:29.951630 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:29.951732 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:29.953207 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:29.954081 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:29.955391 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:29.955825 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:29.955966 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:29.958793 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:29.958799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:29.958806 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:29.958847 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:29.961622 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:29.962436 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:29.962440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:29.962442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:29.962560 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:29.963005 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:29.963012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:29.963019 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:29.963313 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:29.963596 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:29.963961 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 01:11:29.964161 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:11:29.968848 18652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:11:29.975977 18652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:11:30.009702 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:29.989619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.0.97.1:34747)
0619 01:11:29.990366 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:29.990377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:29.999817 (+ 9440us) server_negotiation.cc:407] Connection header received
0619 01:11:30.000247 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:30.000255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:30.000539 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:30.000708 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:30.002547 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.004198 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.006659 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.008229 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.008370 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.008626 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:30.009140 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 01:11:30.009359 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":597,"threads_started":1}
I20260619 01:11:30.009799 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:29.990061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34747 (local address 127.0.0.1:44450)
0619 01:11:29.999610 (+ 9549us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:29.999677 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:29.999981 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:30.001062 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:30.001069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:30.001114 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:30.002289 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:30.002303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.004342 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.004348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:30.006489 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:30.006504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.008752 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.008756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:30.008893 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.008931 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 01:11:30.009081 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 01:11:30.009283 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9385,"mutex_wait_us":554,"spinlock_wait_cycles":3456,"thread_start_us":9080,"threads_started":1}
I20260619 01:11:30.010644 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:29.989266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40129 (local address 127.0.0.1:55214)
0619 01:11:29.999610 (+ 10344us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:29.999677 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:29.999981 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:30.001062 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:30.001069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:30.001108 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:30.002006 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:30.002022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.003959 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.003965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:30.006526 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:30.006544 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.009506 (+ 2962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.009513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:30.009701 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.009753 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 01:11:30.010047 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 01:11:30.010208 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9995,"mutex_wait_us":115,"spinlock_wait_cycles":16512,"thread_start_us":9955,"threads_started":1}
I20260619 01:11:30.010666 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:29.989942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.0.97.3:40129)
0619 01:11:29.999106 (+ 9164us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:29.999118 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:29.999813 (+ 695us) server_negotiation.cc:407] Connection header received
0619 01:11:30.000246 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:30.000254 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:30.000535 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:30.000709 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:30.002238 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.003753 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.006773 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.008250 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.008373 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.008639 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:30.010146 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0619 01:11:30.010341 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8950,"thread_start_us":8772,"threads_started":1}
I20260619 01:11:30.011489 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:30.000229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34009 (local address 127.0.0.1:56548)
0619 01:11:30.002177 (+ 1948us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:30.002196 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:30.002313 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:30.003298 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:30.003304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:30.003330 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:30.003681 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:30.003693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.005779 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.005786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:30.007894 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:30.007906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.010168 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.010173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:30.010345 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.010378 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 01:11:30.010475 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 01:11:30.010618 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1762,"mutex_wait_us":24,"thread_start_us":1353,"threads_started":1}
I20260619 01:11:30.012502 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:30.000483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.0.97.2:34009)
0619 01:11:30.002625 (+ 2142us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:30.002635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:30.002666 (+ 31us) server_negotiation.cc:407] Connection header received
0619 01:11:30.002760 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:30.002768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:30.003017 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:30.003204 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:30.003898 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.005613 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.008082 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.009646 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.009783 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.011894 (+ 2111us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:30.011981 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 01:11:30.012154 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1973,"thread_start_us":1229,"threads_started":1}
I20260619 01:11:30.014249 19045 tablet_service.cc:1467] Processing CreateTablet for tablet 8d26543a1e624613bbfdcb9f88e62868 (DEFAULT_TABLE table=test-table [id=023df2ca36bd4f488f758ecfa5ff05dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:30.014256 18777 tablet_service.cc:1467] Processing CreateTablet for tablet 8d26543a1e624613bbfdcb9f88e62868 (DEFAULT_TABLE table=test-table [id=023df2ca36bd4f488f758ecfa5ff05dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:30.015632 19045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d26543a1e624613bbfdcb9f88e62868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:30.015734 18777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d26543a1e624613bbfdcb9f88e62868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:30.015825 18911 tablet_service.cc:1467] Processing CreateTablet for tablet 8d26543a1e624613bbfdcb9f88e62868 (DEFAULT_TABLE table=test-table [id=023df2ca36bd4f488f758ecfa5ff05dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:30.017151 18911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d26543a1e624613bbfdcb9f88e62868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:30.035920 19129 tablet_bootstrap.cc:492] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: Bootstrap starting.
I20260619 01:11:30.040109 19130 tablet_bootstrap.cc:492] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed: Bootstrap starting.
I20260619 01:11:30.040303 19129 tablet_bootstrap.cc:654] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:30.043037 19129 log.cc:824] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:30.045256 19131 tablet_bootstrap.cc:492] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: Bootstrap starting.
I20260619 01:11:30.048211 19130 tablet_bootstrap.cc:654] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:30.049665 19131 tablet_bootstrap.cc:654] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:30.049896 19129 tablet_bootstrap.cc:492] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: No bootstrap required, opened a new log
I20260619 01:11:30.050100 19130 log.cc:824] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:30.050237 19129 ts_tablet_manager.cc:1397] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260619 01:11:30.051026 19131 log.cc:824] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:30.053907 19130 tablet_bootstrap.cc:492] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed: No bootstrap required, opened a new log
I20260619 01:11:30.054234 19130 ts_tablet_manager.cc:1397] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260619 01:11:30.064998 19131 tablet_bootstrap.cc:492] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: No bootstrap required, opened a new log
I20260619 01:11:30.064752 19129 raft_consensus.cc:348] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609b2e6290543cfbc36e5da255f6aed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34747 } } peers { permanent_uuid: "9aebe42bba7a4f8ab82037a87258d220" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40129 } } peers { permanent_uuid: "50a60bb832e3487c91a7e2c6d913f15d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34009 } }
I20260619 01:11:30.065325 19129 raft_consensus.cc:374] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:30.065443 19129 raft_consensus.cc:729] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aebe42bba7a4f8ab82037a87258d220, State: Initialized, Role: FOLLOWER
I20260619 01:11:30.065699 19131 ts_tablet_manager.cc:1397] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.012s
I20260619 01:11:30.066329 19129 consensus_queue.cc:260] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609b2e6290543cfbc36e5da255f6aed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34747 } } peers { permanent_uuid: "9aebe42bba7a4f8ab82037a87258d220" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40129 } } peers { permanent_uuid: "50a60bb832e3487c91a7e2c6d913f15d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34009 } }
I20260619 01:11:30.067623 19110 heartbeater.cc:503] Master 127.0.97.62:37159 was elected leader, sending a full tablet report...
I20260619 01:11:30.067386 19130 raft_consensus.cc:348] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609b2e6290543cfbc36e5da255f6aed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34747 } } peers { permanent_uuid: "9aebe42bba7a4f8ab82037a87258d220" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40129 } } peers { permanent_uuid: "50a60bb832e3487c91a7e2c6d913f15d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34009 } }
I20260619 01:11:30.067893 19130 raft_consensus.cc:374] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:30.068019 19130 raft_consensus.cc:729] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6609b2e6290543cfbc36e5da255f6aed, State: Initialized, Role: FOLLOWER
I20260619 01:11:30.068794 19130 consensus_queue.cc:260] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609b2e6290543cfbc36e5da255f6aed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34747 } } peers { permanent_uuid: "9aebe42bba7a4f8ab82037a87258d220" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40129 } } peers { permanent_uuid: "50a60bb832e3487c91a7e2c6d913f15d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34009 } }
I20260619 01:11:30.075379 19129 ts_tablet_manager.cc:1428] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: Time spent starting tablet: real 0.025s user 0.017s sys 0.000s
I20260619 01:11:30.078862 19130 ts_tablet_manager.cc:1428] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed: Time spent starting tablet: real 0.025s user 0.014s sys 0.008s
I20260619 01:11:30.078832 19131 raft_consensus.cc:348] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609b2e6290543cfbc36e5da255f6aed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34747 } } peers { permanent_uuid: "9aebe42bba7a4f8ab82037a87258d220" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40129 } } peers { permanent_uuid: "50a60bb832e3487c91a7e2c6d913f15d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34009 } }
I20260619 01:11:30.079494 19131 raft_consensus.cc:374] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:30.079615 19131 raft_consensus.cc:729] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a60bb832e3487c91a7e2c6d913f15d, State: Initialized, Role: FOLLOWER
I20260619 01:11:30.080380 19131 consensus_queue.cc:260] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609b2e6290543cfbc36e5da255f6aed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34747 } } peers { permanent_uuid: "9aebe42bba7a4f8ab82037a87258d220" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40129 } } peers { permanent_uuid: "50a60bb832e3487c91a7e2c6d913f15d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34009 } }
I20260619 01:11:30.081691 18976 heartbeater.cc:503] Master 127.0.97.62:37159 was elected leader, sending a full tablet report...
I20260619 01:11:30.082544 19131 ts_tablet_manager.cc:1428] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260619 01:11:30.105726 19135 raft_consensus.cc:482] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:11:30.106451 19135 raft_consensus.cc:504] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609b2e6290543cfbc36e5da255f6aed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34747 } } peers { permanent_uuid: "9aebe42bba7a4f8ab82037a87258d220" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40129 } } peers { permanent_uuid: "50a60bb832e3487c91a7e2c6d913f15d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34009 } }
I20260619 01:11:30.108239 19135 leader_election.cc:288] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6609b2e6290543cfbc36e5da255f6aed (127.0.97.1:34747), 50a60bb832e3487c91a7e2c6d913f15d (127.0.97.2:34009)
I20260619 01:11:30.114809 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:30.108567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34747 (local address 127.0.97.3:40829)
0619 01:11:30.108783 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:30.108805 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:30.108952 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:30.109611 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:30.109616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:30.109650 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:30.109827 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:30.109836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.111086 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.111092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:30.112963 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:30.112979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.114219 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.114223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:30.114356 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.114381 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 01:11:30.114475 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 01:11:30.114583 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":1}
I20260619 01:11:30.114934 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:30.108618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40829 (local address 127.0.97.1:34747)
0619 01:11:30.108853 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:30.108860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:30.108892 (+ 32us) server_negotiation.cc:407] Connection header received
0619 01:11:30.109150 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:30.109159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:30.109264 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:30.109401 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:30.109989 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.110920 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.113160 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.114029 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.114210 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.114326 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:30.114515 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 01:11:30.114685 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 01:11:30.116389 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:30.109160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57365 (local address 127.0.97.2:34009)
0619 01:11:30.109420 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:30.109428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:30.110006 (+ 578us) server_negotiation.cc:407] Connection header received
0619 01:11:30.110300 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:30.110306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:30.110387 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:30.110487 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:30.111219 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.112307 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.114142 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.115495 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.115630 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.115701 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:30.116099 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 01:11:30.116201 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 01:11:30.116132 18797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d26543a1e624613bbfdcb9f88e62868" candidate_uuid: "9aebe42bba7a4f8ab82037a87258d220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6609b2e6290543cfbc36e5da255f6aed" is_pre_election: true
I20260619 01:11:30.116577 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:30.109005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34009 (local address 127.0.97.3:57365)
0619 01:11:30.109885 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:30.109906 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:30.110118 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:30.110539 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:30.110545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:30.110576 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:30.111006 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:30.111021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.112503 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.112508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:30.113963 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:30.113978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.115734 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.115740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:30.115915 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.115964 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 01:11:30.116069 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:11:30.116218 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":473,"threads_started":1}
I20260619 01:11:30.116847 18797 raft_consensus.cc:2454] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aebe42bba7a4f8ab82037a87258d220 in term 0.
I20260619 01:11:30.117739 18998 leader_election.cc:302] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6609b2e6290543cfbc36e5da255f6aed, 9aebe42bba7a4f8ab82037a87258d220; no voters:
I20260619 01:11:30.117669 18931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d26543a1e624613bbfdcb9f88e62868" candidate_uuid: "9aebe42bba7a4f8ab82037a87258d220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50a60bb832e3487c91a7e2c6d913f15d" is_pre_election: true
I20260619 01:11:30.118203 19135 raft_consensus.cc:2781] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:11:30.118268 18931 raft_consensus.cc:2454] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aebe42bba7a4f8ab82037a87258d220 in term 0.
I20260619 01:11:30.118371 19135 raft_consensus.cc:482] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:11:30.118485 19135 raft_consensus.cc:3037] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:30.120924 19135 raft_consensus.cc:504] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609b2e6290543cfbc36e5da255f6aed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34747 } } peers { permanent_uuid: "9aebe42bba7a4f8ab82037a87258d220" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40129 } } peers { permanent_uuid: "50a60bb832e3487c91a7e2c6d913f15d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34009 } }
I20260619 01:11:30.121745 19135 leader_election.cc:288] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [CANDIDATE]: Term 1 election: Requested vote from peers 6609b2e6290543cfbc36e5da255f6aed (127.0.97.1:34747), 50a60bb832e3487c91a7e2c6d913f15d (127.0.97.2:34009)
I20260619 01:11:30.122385 18797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d26543a1e624613bbfdcb9f88e62868" candidate_uuid: "9aebe42bba7a4f8ab82037a87258d220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6609b2e6290543cfbc36e5da255f6aed"
I20260619 01:11:30.122483 18931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d26543a1e624613bbfdcb9f88e62868" candidate_uuid: "9aebe42bba7a4f8ab82037a87258d220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50a60bb832e3487c91a7e2c6d913f15d"
I20260619 01:11:30.122620 18797 raft_consensus.cc:3037] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:30.122769 18931 raft_consensus.cc:3037] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d [term 0 FOLLOWER]: Advancing to term 1
W20260619 01:11:30.124591 18977 tablet.cc:2369] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:11:30.125478 18931 raft_consensus.cc:2454] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aebe42bba7a4f8ab82037a87258d220 in term 1.
I20260619 01:11:30.125648 18797 raft_consensus.cc:2454] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aebe42bba7a4f8ab82037a87258d220 in term 1.
I20260619 01:11:30.125986 18998 leader_election.cc:302] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50a60bb832e3487c91a7e2c6d913f15d, 9aebe42bba7a4f8ab82037a87258d220; no voters:
I20260619 01:11:30.126328 19135 raft_consensus.cc:2781] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:30.126788 19135 raft_consensus.cc:686] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 1 LEADER]: Becoming Leader. State: Replica: 9aebe42bba7a4f8ab82037a87258d220, State: Running, Role: LEADER
I20260619 01:11:30.127295 19135 consensus_queue.cc:237] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609b2e6290543cfbc36e5da255f6aed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34747 } } peers { permanent_uuid: "9aebe42bba7a4f8ab82037a87258d220" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40129 } } peers { permanent_uuid: "50a60bb832e3487c91a7e2c6d913f15d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34009 } }
I20260619 01:11:30.132395 18652 catalog_manager.cc:5495] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9aebe42bba7a4f8ab82037a87258d220 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9aebe42bba7a4f8ab82037a87258d220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6609b2e6290543cfbc36e5da255f6aed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aebe42bba7a4f8ab82037a87258d220" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50a60bb832e3487c91a7e2c6d913f15d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34009 } health_report { overall_health: UNKNOWN } } }
W20260619 01:11:30.161113 19111 tablet.cc:2369] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:11:30.174104 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:30.168066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.0.97.3:40129)
0619 01:11:30.168287 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:30.168295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:30.168403 (+ 108us) server_negotiation.cc:407] Connection header received
0619 01:11:30.168684 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:30.168688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:30.168779 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:30.168948 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:11:30.169417 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.170175 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.171607 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.172056 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.172192 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.172850 (+ 658us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:11:30.173460 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:11:30.173648 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:30.173800 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 01:11:30.173859 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:11:30.183171 18797 raft_consensus.cc:1261] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed [term 1 FOLLOWER]: Refusing update from remote peer 9aebe42bba7a4f8ab82037a87258d220: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:30.183223 18931 raft_consensus.cc:1261] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d [term 1 FOLLOWER]: Refusing update from remote peer 9aebe42bba7a4f8ab82037a87258d220: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:30.184171 19135 consensus_queue.cc:1035] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6609b2e6290543cfbc36e5da255f6aed" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:30.184546 19139 consensus_queue.cc:1035] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50a60bb832e3487c91a7e2c6d913f15d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:30.203122 19143 mvcc.cc:205] Tried to move back new op lower bound from 7298381783779184640 to 7298381783565496320. Current Snapshot: MvccSnapshot[applied={T|T < 7298381783779184640}]
I20260619 01:11:30.217120 18653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:30.217464 18653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:30.221356 18653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b88f98d0a87444dab8cc8007ea81c7a: Sending DeleteTablet for 3 replicas of tablet 8d26543a1e624613bbfdcb9f88e62868
I20260619 01:11:30.222749 19045 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d26543a1e624613bbfdcb9f88e62868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55214
I20260619 01:11:30.222774 18777 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d26543a1e624613bbfdcb9f88e62868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44450
I20260619 01:11:30.223050 18911 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d26543a1e624613bbfdcb9f88e62868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56548
I20260619 01:11:30.226985 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 18714
I20260619 01:11:30.235559 19153 tablet_replica.cc:330] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed: stopping tablet replica
I20260619 01:11:30.237035 19152 tablet_replica.cc:330] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: stopping tablet replica
I20260619 01:11:30.237345 19154 tablet_replica.cc:330] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: stopping tablet replica
I20260619 01:11:30.237372 19153 raft_consensus.cc:2229] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:30.237952 19154 raft_consensus.cc:2229] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:30.238059 19153 raft_consensus.cc:2258] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:30.238489 19152 raft_consensus.cc:2229] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:11:30.239140 19154 raft_consensus.cc:2258] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:30.239894 19152 raft_consensus.cc:2258] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:30.240674 19153 ts_tablet_manager.cc:1905] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:30.243366 19152 ts_tablet_manager.cc:1905] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:30.245314 19154 ts_tablet_manager.cc:1905] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:30.248878 19153 ts_tablet_manager.cc:1918] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:30.249068 19153 log.cc:1196] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/8d26543a1e624613bbfdcb9f88e62868
I20260619 01:11:30.249698 19153 ts_tablet_manager.cc:1939] T 8d26543a1e624613bbfdcb9f88e62868 P 6609b2e6290543cfbc36e5da255f6aed: Deleting consensus metadata
I20260619 01:11:30.251773 18637 catalog_manager.cc:4841] TS 6609b2e6290543cfbc36e5da255f6aed (127.0.97.1:34747): tablet 8d26543a1e624613bbfdcb9f88e62868 (table test-table [id=023df2ca36bd4f488f758ecfa5ff05dc]) successfully deleted
Jun 19 01:11:30 dist-test-slave-5p8w krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:11:30.254747 19154 ts_tablet_manager.cc:1918] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:30.255012 19154 log.cc:1196] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/8d26543a1e624613bbfdcb9f88e62868
I20260619 01:11:30.255638 19154 ts_tablet_manager.cc:1939] T 8d26543a1e624613bbfdcb9f88e62868 P 50a60bb832e3487c91a7e2c6d913f15d: Deleting consensus metadata
I20260619 01:11:30.257750 18637 catalog_manager.cc:4841] TS 50a60bb832e3487c91a7e2c6d913f15d (127.0.97.2:34009): tablet 8d26543a1e624613bbfdcb9f88e62868 (table test-table [id=023df2ca36bd4f488f758ecfa5ff05dc]) successfully deleted
I20260619 01:11:30.258040 19152 ts_tablet_manager.cc:1918] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:30.258285 19152 log.cc:1196] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/8d26543a1e624613bbfdcb9f88e62868
I20260619 01:11:30.258934 19152 ts_tablet_manager.cc:1939] T 8d26543a1e624613bbfdcb9f88e62868 P 9aebe42bba7a4f8ab82037a87258d220: Deleting consensus metadata
I20260619 01:11:30.261001 18639 catalog_manager.cc:4841] TS 9aebe42bba7a4f8ab82037a87258d220 (127.0.97.3:40129): tablet 8d26543a1e624613bbfdcb9f88e62868 (table test-table [id=023df2ca36bd4f488f758ecfa5ff05dc]) successfully deleted
I20260619 01:11:30.263574 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:30.228846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.0.97.1:34747)
0619 01:11:30.229023 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:30.229030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:30.239103 (+ 10073us) server_negotiation.cc:407] Connection header received
0619 01:11:30.239550 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:30.239556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:30.239658 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:30.239754 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:30.241967 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.243324 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.245748 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.246499 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.246664 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.255778 (+ 9114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:30.255797 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:30.255818 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:30.255879 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:30.260251 (+ 4372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:30.261362 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:30.261371 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:30.261374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:30.261568 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:30.262079 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:30.262083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:30.262085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:30.262472 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:30.262732 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:30.263080 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 01:11:30.263258 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 01:11:30.264325 18837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:30.406643 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 18714
I20260619 01:11:30.425616 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 18848
Jun 19 01:11:30 dist-test-slave-5p8w krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:11:30.443655 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:30.426731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.0.97.2:34009)
0619 01:11:30.426951 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:30.426959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:30.427002 (+ 43us) server_negotiation.cc:407] Connection header received
0619 01:11:30.427074 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:30.427080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:30.427182 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:30.427297 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:30.428518 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.429834 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.431197 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.431841 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.431988 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.437183 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:30.437203 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:30.437221 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:30.437349 (+ 128us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:30.440430 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:30.441526 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:30.441535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:30.441537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:30.441683 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:30.442101 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:30.442104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:30.442106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:30.442535 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:30.442751 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:30.443140 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 01:11:30.443316 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 01:11:30.444561 18971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:30.538130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 18848
I20260619 01:11:30.556437 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 18982
Jun 19 01:11:30 dist-test-slave-5p8w krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:11:30.576176 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:30.557487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.0.97.3:40129)
0619 01:11:30.557660 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:30.557664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:30.557683 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:11:30.557817 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:30.557821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:30.557888 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:30.557975 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:30.559834 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.560704 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.562877 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:30.563341 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:30.563443 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:30.569502 (+ 6059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:30.569521 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:30.569538 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:30.569585 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:30.572850 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:30.573945 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:30.573953 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:30.573955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:30.574125 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:30.574566 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:30.574570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:30.574572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:30.574947 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:30.575192 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:30.575620 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 01:11:30.575810 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:11:30.577054 19105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:30.703534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 18982
I20260619 01:11:30.723568 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 18621
I20260619 01:11:30.724681 18683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:30.836877 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 18621
2026-06-19T01:11:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4476 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T01:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:30Z Disabled control of system clock
I20260619 01:11:30.916900 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34535
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:11:31.272174 19163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:31.272506 19163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:31.272679 19163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:31.280944 19163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:31.281103 19163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:31.281173 19163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:31.281270 19163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:31.281368 19163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:31.292996 19163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:31.294847 19163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:31.297693 19163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:31.313649 19168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:31.313932 19171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:31.313678 19169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:31.316474 19163 server_base.cc:1034] running on GCE node
I20260619 01:11:31.317734 19163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:31.320056 19163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:31.321359 19163 hybrid_clock.cc:648] HybridClock initialized: now 1781831491321266 us; error 56 us; skew 500 ppm
I20260619 01:11:31.321873 19163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:31.325362 19163 webserver.cc:460] Webserver started at http://127.0.97.62:37757/ using document root <none> and password file <none>
I20260619 01:11:31.326155 19163 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:31.326290 19163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:31.326635 19163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:31.330081 19163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "53a685c457cb411c8e23b0d50cfb362d"
format_stamp: "Formatted at 2026-06-19 01:11:31 on dist-test-slave-5p8w"
server_key: "feec895ce48c2e79e9c4d67c743bd486"
server_key_iv: "c9cea86b07ccdd63943723362385fe8f"
server_key_version: "encryptionkey@0"
I20260619 01:11:31.331233 19163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "53a685c457cb411c8e23b0d50cfb362d"
format_stamp: "Formatted at 2026-06-19 01:11:31 on dist-test-slave-5p8w"
server_key: "feec895ce48c2e79e9c4d67c743bd486"
server_key_iv: "c9cea86b07ccdd63943723362385fe8f"
server_key_version: "encryptionkey@0"
I20260619 01:11:31.339177 19163 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260619 01:11:31.343206 19177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:31.344872 19163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:11:31.345096 19163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "53a685c457cb411c8e23b0d50cfb362d"
format_stamp: "Formatted at 2026-06-19 01:11:31 on dist-test-slave-5p8w"
server_key: "feec895ce48c2e79e9c4d67c743bd486"
server_key_iv: "c9cea86b07ccdd63943723362385fe8f"
server_key_version: "encryptionkey@0"
I20260619 01:11:31.345439 19163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:31.363327 19163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:31.364203 19163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:31.364466 19163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:31.384793 19163 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34535
I20260619 01:11:31.384800 19228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34535 every 8 connection(s)
I20260619 01:11:31.386330 19163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:31.390496 19229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:31.392979 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 19163
I20260619 01:11:31.393172 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:31.393760 388 external_mini_cluster.cc:1428] Setting key d4c6a376cea60453c3eefc565e11feac
I20260619 01:11:31.404640 19229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d: Bootstrap starting.
I20260619 01:11:31.408255 19229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:31.409510 19229 log.cc:824] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:31.409817 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:31.396025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.0.97.62:34535)
0619 01:11:31.397256 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:31.397276 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:31.397421 (+ 145us) server_negotiation.cc:407] Connection header received
0619 01:11:31.400194 (+ 2773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:31.400228 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:31.400878 (+ 650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:31.401372 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:31.402000 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:31.403324 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:31.404765 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:31.405619 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:31.405835 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:31.407139 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:31.407147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:31.407163 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:31.407760 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:31.407855 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:31.407879 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:31.408029 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:31.408278 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 01:11:31.408671 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":419,"threads_started":1}
I20260619 01:11:31.412863 19229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d: No bootstrap required, opened a new log
I20260619 01:11:31.421505 19229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a685c457cb411c8e23b0d50cfb362d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34535 } }
I20260619 01:11:31.421972 19229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:31.422078 19229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a685c457cb411c8e23b0d50cfb362d, State: Initialized, Role: FOLLOWER
I20260619 01:11:31.422833 19229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a685c457cb411c8e23b0d50cfb362d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34535 } }
I20260619 01:11:31.423089 19229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:31.423241 19229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:31.423452 19229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:31.426011 19229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a685c457cb411c8e23b0d50cfb362d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34535 } }
I20260619 01:11:31.426666 19229 leader_election.cc:302] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53a685c457cb411c8e23b0d50cfb362d; no voters:
I20260619 01:11:31.427395 19229 leader_election.cc:288] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:31.427630 19234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:31.429088 19234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [term 1 LEADER]: Becoming Leader. State: Replica: 53a685c457cb411c8e23b0d50cfb362d, State: Running, Role: LEADER
I20260619 01:11:31.430064 19229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:31.430068 19234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a685c457cb411c8e23b0d50cfb362d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34535 } }
I20260619 01:11:31.435279 19235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53a685c457cb411c8e23b0d50cfb362d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a685c457cb411c8e23b0d50cfb362d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34535 } } }
I20260619 01:11:31.435279 19236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53a685c457cb411c8e23b0d50cfb362d. Latest consensus state: current_term: 1 leader_uuid: "53a685c457cb411c8e23b0d50cfb362d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a685c457cb411c8e23b0d50cfb362d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34535 } } }
I20260619 01:11:31.435755 19235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:31.435874 19236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:31.438619 19244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:31.442924 19244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:31.452435 19244 catalog_manager.cc:1347] Generated new cluster ID: 83fc6bf952db4077ac0bc75f9a047639
I20260619 01:11:31.452585 19244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:31.470937 19244 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:31.472354 19244 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:31.484593 19244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d: Generated new TSK 0
I20260619 01:11:31.485599 19244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:31.499583 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34535
--builtin_ntp_servers=127.0.97.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:11:31.850883 19253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:31.851166 19253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:31.851382 19253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:31.859035 19253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:31.859153 19253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:31.859368 19253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:11:31.868934 19253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:31.870373 19253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:31.872217 19253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:31.885862 19258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:31.885907 19261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:31.886838 19253 server_base.cc:1034] running on GCE node
W20260619 01:11:31.887058 19259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:31.887844 19253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:31.888863 19253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:31.890100 19253 hybrid_clock.cc:648] HybridClock initialized: now 1781831491890033 us; error 63 us; skew 500 ppm
I20260619 01:11:31.890453 19253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:31.892809 19253 webserver.cc:460] Webserver started at http://127.0.97.1:44801/ using document root <none> and password file <none>
I20260619 01:11:31.893688 19253 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:31.893824 19253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:31.894163 19253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:31.897190 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "1ccd94aeec0b46368c41a0c9284ed013"
format_stamp: "Formatted at 2026-06-19 01:11:31 on dist-test-slave-5p8w"
server_key: "291b7d8f100b0f6a1f5fe1f7426b4ccd"
server_key_iv: "89ae52a5f374c2ac20f9ac8e98304ae5"
server_key_version: "encryptionkey@0"
I20260619 01:11:31.898090 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1ccd94aeec0b46368c41a0c9284ed013"
format_stamp: "Formatted at 2026-06-19 01:11:31 on dist-test-slave-5p8w"
server_key: "291b7d8f100b0f6a1f5fe1f7426b4ccd"
server_key_iv: "89ae52a5f374c2ac20f9ac8e98304ae5"
server_key_version: "encryptionkey@0"
I20260619 01:11:31.903376 19253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:11:31.907169 19267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:31.908752 19253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:11:31.908941 19253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "1ccd94aeec0b46368c41a0c9284ed013"
format_stamp: "Formatted at 2026-06-19 01:11:31 on dist-test-slave-5p8w"
server_key: "291b7d8f100b0f6a1f5fe1f7426b4ccd"
server_key_iv: "89ae52a5f374c2ac20f9ac8e98304ae5"
server_key_version: "encryptionkey@0"
I20260619 01:11:31.909181 19253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:31.924578 19253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:31.925554 19253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:31.925841 19253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:31.926869 19253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:31.928432 19253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:31.928519 19253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:31.928676 19253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:31.928773 19253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:31.964046 19253 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44961
I20260619 01:11:31.964103 19379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44961 every 8 connection(s)
I20260619 01:11:31.965592 19253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:11:31.967764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 19253
I20260619 01:11:31.967957 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:11:31.968477 388 external_mini_cluster.cc:1428] Setting key 033157a53a2125403575cbdd684166e7
I20260619 01:11:31.971312 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34535
--builtin_ntp_servers=127.0.97.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:11:31.981274 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:31.968644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55521 (local address 127.0.97.62:34535)
0619 01:11:31.969136 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:31.969141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:31.970753 (+ 1612us) server_negotiation.cc:407] Connection header received
0619 01:11:31.972689 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:31.972697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:31.972788 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:31.972911 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:31.974115 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:31.975292 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:31.977794 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:31.978549 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:31.978705 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:31.980489 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:31.980494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:31.980497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:31.980519 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:31.980563 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:31.980575 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:31.980643 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:31.980919 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 01:11:31.981013 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":234,"threads_started":1}
I20260619 01:11:31.982837 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:31.969450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34535 (local address 127.0.97.1:55521)
0619 01:11:31.970589 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:31.970634 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:31.972407 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:31.973123 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:31.973132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:31.973229 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:31.973933 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:31.973952 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:31.975487 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:31.975494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:31.977599 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:31.977616 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:31.978782 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:31.978787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:31.979173 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:31.980077 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:11:31.980082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:31.980146 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:11:31.980263 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:31.980671 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:31.980685 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:31.980697 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 01:11:31.980896 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 01:11:31.981392 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":376,"threads_started":1}
I20260619 01:11:31.984827 19380 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34535
I20260619 01:11:31.985199 19380 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:31.986064 19380 heartbeater.cc:511] Master 127.0.97.62:34535 requested a full tablet report, sending...
I20260619 01:11:31.988531 19194 ts_manager.cc:194] Registered new tserver with Master: 1ccd94aeec0b46368c41a0c9284ed013 (127.0.97.1:44961)
I20260619 01:11:31.990834 19194 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:55521
W20260619 01:11:32.324364 19384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:32.324692 19384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:32.324786 19384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:32.332741 19384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:32.332881 19384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:32.333001 19384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:11:32.343071 19384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:32.344455 19384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:32.346421 19384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:32.361014 19389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:32.360956 19390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:32.360998 19392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:32.362092 19384 server_base.cc:1034] running on GCE node
I20260619 01:11:32.362812 19384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:32.363955 19384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:32.365211 19384 hybrid_clock.cc:648] HybridClock initialized: now 1781831492365128 us; error 85 us; skew 500 ppm
I20260619 01:11:32.365620 19384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:32.368254 19384 webserver.cc:460] Webserver started at http://127.0.97.2:32815/ using document root <none> and password file <none>
I20260619 01:11:32.369081 19384 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:32.369227 19384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:32.369611 19384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:32.372916 19384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "7d7a9178640e4524ae5c34739f77e198"
format_stamp: "Formatted at 2026-06-19 01:11:32 on dist-test-slave-5p8w"
server_key: "2e241235a1d41b9e27a6aa3f7fdb0104"
server_key_iv: "893f14b3de22b8dfc14ec63c0b6bdfe7"
server_key_version: "encryptionkey@0"
I20260619 01:11:32.373869 19384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7d7a9178640e4524ae5c34739f77e198"
format_stamp: "Formatted at 2026-06-19 01:11:32 on dist-test-slave-5p8w"
server_key: "2e241235a1d41b9e27a6aa3f7fdb0104"
server_key_iv: "893f14b3de22b8dfc14ec63c0b6bdfe7"
server_key_version: "encryptionkey@0"
I20260619 01:11:32.379612 19384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:11:32.383626 19398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:32.385524 19384 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:11:32.385725 19384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "7d7a9178640e4524ae5c34739f77e198"
format_stamp: "Formatted at 2026-06-19 01:11:32 on dist-test-slave-5p8w"
server_key: "2e241235a1d41b9e27a6aa3f7fdb0104"
server_key_iv: "893f14b3de22b8dfc14ec63c0b6bdfe7"
server_key_version: "encryptionkey@0"
I20260619 01:11:32.386001 19384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:32.410987 19384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:32.411885 19384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:32.412159 19384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:32.413218 19384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:32.414974 19384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:32.415063 19384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:32.415221 19384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:32.415315 19384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:32.450075 19384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40801
I20260619 01:11:32.450088 19510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40801 every 8 connection(s)
I20260619 01:11:32.451711 19384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:11:32.453694 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 19384
I20260619 01:11:32.453898 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:11:32.454460 388 external_mini_cluster.cc:1428] Setting key 040e381f8bfe31b40d8c801555f12b2e
I20260619 01:11:32.457798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34535
--builtin_ntp_servers=127.0.97.20:44377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:11:32.466193 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:32.454557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37079 (local address 127.0.97.62:34535)
0619 01:11:32.454763 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:32.454767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:32.456721 (+ 1954us) server_negotiation.cc:407] Connection header received
0619 01:11:32.458775 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:32.458784 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:32.458882 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:32.459009 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:32.460253 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.461439 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.463071 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.463789 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.463934 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:32.465384 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:32.465393 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:32.465400 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:32.465426 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:32.465466 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:32.465487 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:32.465561 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:32.465836 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 01:11:32.465941 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 01:11:32.467760 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:32.455273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34535 (local address 127.0.97.2:37079)
0619 01:11:32.456548 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:32.456600 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:32.458479 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:32.459330 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:32.459340 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:32.459450 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:32.460066 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:32.460083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.461591 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.461594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:32.462890 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:32.462903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.464018 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.464023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:32.464243 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:32.465006 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:11:32.465009 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:32.465052 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:11:32.465150 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:32.465587 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:32.465599 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:32.465610 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 01:11:32.465821 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 01:11:32.466307 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":381,"threads_started":1}
I20260619 01:11:32.469349 19511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34535
I20260619 01:11:32.469697 19511 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:32.470506 19511 heartbeater.cc:511] Master 127.0.97.62:34535 requested a full tablet report, sending...
I20260619 01:11:32.472065 19194 ts_manager.cc:194] Registered new tserver with Master: 7d7a9178640e4524ae5c34739f77e198 (127.0.97.2:40801)
I20260619 01:11:32.472903 19194 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:37079
W20260619 01:11:32.795951 19514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:32.796262 19514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:32.796442 19514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:32.803833 19514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:32.803943 19514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:32.804126 19514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:11:32.813659 19514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:32.815097 19514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:32.816941 19514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:32.830505 19522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:32.830587 19520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:32.830758 19519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:32.831501 19514 server_base.cc:1034] running on GCE node
I20260619 01:11:32.832176 19514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:32.833204 19514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:32.834463 19514 hybrid_clock.cc:648] HybridClock initialized: now 1781831492834403 us; error 74 us; skew 500 ppm
I20260619 01:11:32.834800 19514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:32.837131 19514 webserver.cc:460] Webserver started at http://127.0.97.3:41095/ using document root <none> and password file <none>
I20260619 01:11:32.837913 19514 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:32.838047 19514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:32.838389 19514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:32.841387 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "95b57fc7ee0647429deaa6d3c6449b6a"
format_stamp: "Formatted at 2026-06-19 01:11:32 on dist-test-slave-5p8w"
server_key: "be7fe6386c3ccef525f3bdfbb3f0ebc6"
server_key_iv: "37739dd3e91fcdce93e22ded7af9e899"
server_key_version: "encryptionkey@0"
I20260619 01:11:32.842216 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "95b57fc7ee0647429deaa6d3c6449b6a"
format_stamp: "Formatted at 2026-06-19 01:11:32 on dist-test-slave-5p8w"
server_key: "be7fe6386c3ccef525f3bdfbb3f0ebc6"
server_key_iv: "37739dd3e91fcdce93e22ded7af9e899"
server_key_version: "encryptionkey@0"
I20260619 01:11:32.847370 19514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:11:32.851125 19528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:32.852926 19514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 01:11:32.853106 19514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "95b57fc7ee0647429deaa6d3c6449b6a"
format_stamp: "Formatted at 2026-06-19 01:11:32 on dist-test-slave-5p8w"
server_key: "be7fe6386c3ccef525f3bdfbb3f0ebc6"
server_key_iv: "37739dd3e91fcdce93e22ded7af9e899"
server_key_version: "encryptionkey@0"
I20260619 01:11:32.853374 19514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:32.877182 19514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:32.878172 19514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:32.878427 19514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:32.879415 19514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:32.881002 19514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:32.881140 19514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:32.881285 19514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:32.881405 19514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 01:11:32.915807 19514 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36307
I20260619 01:11:32.915911 19640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36307 every 8 connection(s)
I20260619 01:11:32.917254 19514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:11:32.925370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 19514
I20260619 01:11:32.925573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:11:32.926033 388 external_mini_cluster.cc:1428] Setting key 9455cc124616e4df0fd997d199dac1ec
I20260619 01:11:32.931210 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:32.920282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32867 (local address 127.0.97.62:34535)
0619 01:11:32.920497 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:32.920502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:32.921980 (+ 1478us) server_negotiation.cc:407] Connection header received
0619 01:11:32.923872 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:32.923876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:32.923953 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:32.924074 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:32.925202 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.926157 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.928268 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.928829 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.928981 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:32.930510 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:32.930515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:32.930518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:32.930537 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:32.930569 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:32.930581 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:32.930653 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:32.930871 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 01:11:32.930961 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 01:11:32.932341 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:32.920808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34535 (local address 127.0.97.3:32867)
0619 01:11:32.921812 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:32.921857 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:32.923539 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:32.924301 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:32.924310 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:32.924414 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:32.925008 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:32.925024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.926334 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.926339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:32.928112 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:32.928127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.929094 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.929110 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:32.929405 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:32.930119 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:11:32.930121 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:32.930209 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:11:32.930321 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:32.930667 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:32.930674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:32.930681 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 01:11:32.930850 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 01:11:32.931190 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":301,"threads_started":1}
I20260619 01:11:32.933871 19641 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34535
I20260619 01:11:32.934199 19641 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:32.935006 19641 heartbeater.cc:511] Master 127.0.97.62:34535 requested a full tablet report, sending...
I20260619 01:11:32.936507 19194 ts_manager.cc:194] Registered new tserver with Master: 95b57fc7ee0647429deaa6d3c6449b6a (127.0.97.3:36307)
I20260619 01:11:32.937371 19194 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:32867
I20260619 01:11:32.941377 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:11:32.951344 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:32.945265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.0.97.62:34535)
0619 01:11:32.945528 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:32.945537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:32.945957 (+ 420us) server_negotiation.cc:407] Connection header received
0619 01:11:32.946092 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:32.946095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:32.946170 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:32.946253 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:32.946702 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.947399 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.948766 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.949205 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.949362 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:32.950581 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:32.950586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:32.950589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:32.950608 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:32.950640 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:32.950652 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:32.950760 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:32.950944 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 01:11:32.951027 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 01:11:32.955415 19194 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:11:32.962615 19194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:11:32.987588 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:32.976442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.0.97.3:36307)
0619 01:11:32.977176 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:32.977184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:32.977361 (+ 177us) server_negotiation.cc:407] Connection header received
0619 01:11:32.977688 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:32.977700 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:32.978032 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:32.978234 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:32.979420 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.981525 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.985281 (+ 3756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.986245 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.986404 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:32.986719 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:32.986958 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 01:11:32.987177 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":1691,"threads_started":1}
I20260619 01:11:32.987709 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:32.977370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44961 (local address 127.0.0.1:44968)
0619 01:11:32.978938 (+ 1568us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:32.978962 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:32.979117 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:32.980086 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:32.980090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:32.980116 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:32.980482 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:32.980498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.982693 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.982701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:32.984952 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:32.984968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.986861 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.986870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:32.987041 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:32.987075 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 01:11:32.987161 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 01:11:32.987360 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1394,"mutex_wait_us":59,"thread_start_us":1206,"threads_started":1}
I20260619 01:11:32.987775 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:32.977722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.0.97.1:44961)
0619 01:11:32.979640 (+ 1918us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:32.979646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:32.979669 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:11:32.979747 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:32.979752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:32.979910 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:32.980100 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:32.980990 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.982509 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.985158 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.986607 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.986752 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:32.987039 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:32.987221 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 01:11:32.987391 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1771,"thread_start_us":1645,"threads_started":1}
I20260619 01:11:32.988551 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:32.975555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36307 (local address 127.0.0.1:42808)
0619 01:11:32.977214 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:32.977234 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:32.977466 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:32.978612 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:32.978625 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:32.978663 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:32.979176 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:32.979193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.981681 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.981687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:32.984756 (+ 3069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:32.984770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.986470 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.986476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:32.986683 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:32.986738 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 01:11:32.987870 (+ 1132us) client_negotiation.cc:240] Negotiation successful
0619 01:11:32.988148 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1508,"mutex_wait_us":34,"thread_start_us":1363,"threads_started":1}
I20260619 01:11:32.990463 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:32.975108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40801 (local address 127.0.0.1:43000)
0619 01:11:32.976305 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:32.976353 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:32.976583 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:32.982561 (+ 5978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:32.982565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:32.982590 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:32.983088 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:32.983098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.984386 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.984394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:32.986575 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:32.986589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.989669 (+ 3080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.989678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:32.989846 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:32.989883 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 01:11:32.989985 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 01:11:32.990132 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":2,"thread_start_us":602,"threads_started":1}
I20260619 01:11:32.990635 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:32.976014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.0.97.2:40801)
0619 01:11:32.982119 (+ 6105us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:32.982125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:32.982149 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:11:32.982218 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:32.982223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:32.982382 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:32.982579 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:32.983255 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.984247 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.986788 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:32.989134 (+ 2346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:32.989267 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:32.989539 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:32.990058 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 01:11:32.990234 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5957,"thread_start_us":5829,"threads_started":1}
I20260619 01:11:32.992183 19315 tablet_service.cc:1467] Processing CreateTablet for tablet 5c468d9bb79b49bdba7f43ef3ac1db4e (DEFAULT_TABLE table=test-table [id=23aca0809ca04ff38406231d895f55b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:32.992214 19576 tablet_service.cc:1467] Processing CreateTablet for tablet 5c468d9bb79b49bdba7f43ef3ac1db4e (DEFAULT_TABLE table=test-table [id=23aca0809ca04ff38406231d895f55b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:32.993712 19315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c468d9bb79b49bdba7f43ef3ac1db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:32.993755 19576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c468d9bb79b49bdba7f43ef3ac1db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:32.994771 19446 tablet_service.cc:1467] Processing CreateTablet for tablet 5c468d9bb79b49bdba7f43ef3ac1db4e (DEFAULT_TABLE table=test-table [id=23aca0809ca04ff38406231d895f55b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:32.995841 19380 heartbeater.cc:503] Master 127.0.97.62:34535 was elected leader, sending a full tablet report...
I20260619 01:11:32.996249 19446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c468d9bb79b49bdba7f43ef3ac1db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:33.006170 19659 tablet_bootstrap.cc:492] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013: Bootstrap starting.
I20260619 01:11:33.010298 19659 tablet_bootstrap.cc:654] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:33.010504 19660 tablet_bootstrap.cc:492] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: Bootstrap starting.
I20260619 01:11:33.011897 19659 log.cc:824] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:33.012576 19661 tablet_bootstrap.cc:492] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: Bootstrap starting.
I20260619 01:11:33.014415 19660 tablet_bootstrap.cc:654] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:33.015439 19660 log.cc:824] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:33.016124 19659 tablet_bootstrap.cc:492] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013: No bootstrap required, opened a new log
I20260619 01:11:33.016418 19659 ts_tablet_manager.cc:1397] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 01:11:33.016749 19661 tablet_bootstrap.cc:654] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:33.018064 19661 log.cc:824] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:33.020838 19661 tablet_bootstrap.cc:492] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: No bootstrap required, opened a new log
I20260619 01:11:33.021107 19660 tablet_bootstrap.cc:492] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: No bootstrap required, opened a new log
I20260619 01:11:33.021179 19661 ts_tablet_manager.cc:1397] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 01:11:33.021430 19660 ts_tablet_manager.cc:1397] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260619 01:11:33.027340 19659 raft_consensus.cc:348] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7a9178640e4524ae5c34739f77e198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40801 } } peers { permanent_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36307 } } peers { permanent_uuid: "1ccd94aeec0b46368c41a0c9284ed013" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } }
I20260619 01:11:33.027864 19659 raft_consensus.cc:374] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:33.027971 19659 raft_consensus.cc:729] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccd94aeec0b46368c41a0c9284ed013, State: Initialized, Role: FOLLOWER
I20260619 01:11:33.028698 19659 consensus_queue.cc:260] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7a9178640e4524ae5c34739f77e198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40801 } } peers { permanent_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36307 } } peers { permanent_uuid: "1ccd94aeec0b46368c41a0c9284ed013" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } }
I20260619 01:11:33.030066 19661 raft_consensus.cc:348] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7a9178640e4524ae5c34739f77e198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40801 } } peers { permanent_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36307 } } peers { permanent_uuid: "1ccd94aeec0b46368c41a0c9284ed013" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } }
I20260619 01:11:33.030603 19661 raft_consensus.cc:374] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:33.030712 19661 raft_consensus.cc:729] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7a9178640e4524ae5c34739f77e198, State: Initialized, Role: FOLLOWER
I20260619 01:11:33.031173 19659 ts_tablet_manager.cc:1428] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
I20260619 01:11:33.031463 19661 consensus_queue.cc:260] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7a9178640e4524ae5c34739f77e198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40801 } } peers { permanent_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36307 } } peers { permanent_uuid: "1ccd94aeec0b46368c41a0c9284ed013" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } }
I20260619 01:11:33.032680 19660 raft_consensus.cc:348] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7a9178640e4524ae5c34739f77e198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40801 } } peers { permanent_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36307 } } peers { permanent_uuid: "1ccd94aeec0b46368c41a0c9284ed013" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } }
I20260619 01:11:33.033159 19660 raft_consensus.cc:374] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:33.033279 19660 raft_consensus.cc:729] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b57fc7ee0647429deaa6d3c6449b6a, State: Initialized, Role: FOLLOWER
I20260619 01:11:33.034194 19511 heartbeater.cc:503] Master 127.0.97.62:34535 was elected leader, sending a full tablet report...
I20260619 01:11:33.034153 19660 consensus_queue.cc:260] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7a9178640e4524ae5c34739f77e198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40801 } } peers { permanent_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36307 } } peers { permanent_uuid: "1ccd94aeec0b46368c41a0c9284ed013" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } }
I20260619 01:11:33.034889 19661 ts_tablet_manager.cc:1428] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260619 01:11:33.036015 19641 heartbeater.cc:503] Master 127.0.97.62:34535 was elected leader, sending a full tablet report...
I20260619 01:11:33.036702 19660 ts_tablet_manager.cc:1428] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260619 01:11:33.154183 19665 raft_consensus.cc:482] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:11:33.154460 19665 raft_consensus.cc:504] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7a9178640e4524ae5c34739f77e198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40801 } } peers { permanent_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36307 } } peers { permanent_uuid: "1ccd94aeec0b46368c41a0c9284ed013" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } }
I20260619 01:11:33.156179 19665 leader_election.cc:288] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d7a9178640e4524ae5c34739f77e198 (127.0.97.2:40801), 95b57fc7ee0647429deaa6d3c6449b6a (127.0.97.3:36307)
I20260619 01:11:33.163403 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.156697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36891 (local address 127.0.97.2:40801)
0619 01:11:33.156911 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:33.156919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:33.157795 (+ 876us) server_negotiation.cc:407] Connection header received
0619 01:11:33.158047 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:33.158056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:33.158148 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:33.158254 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:33.158832 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.159524 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.161713 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.162484 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.162636 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:33.162733 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:33.163006 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 01:11:33.163134 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:11:33.163453 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.156662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55891 (local address 127.0.97.3:36307)
0619 01:11:33.156967 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:33.156977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:33.157737 (+ 760us) server_negotiation.cc:407] Connection header received
0619 01:11:33.158011 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:33.158015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:33.158076 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:33.158177 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:33.158815 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.159537 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.161773 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.162571 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.162720 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:33.162823 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:33.163082 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 01:11:33.163209 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260619 01:11:33.163647 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.156688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40801 (local address 127.0.97.1:36891)
0619 01:11:33.157674 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:33.157693 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:33.157874 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:33.158275 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:33.158279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:33.158300 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:33.158688 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:33.158698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.159670 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.159682 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:33.161530 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:33.161542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.162685 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.162691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:33.162844 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:33.162868 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 01:11:33.162978 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 01:11:33.163117 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":81,"thread_start_us":637,"threads_started":1}
I20260619 01:11:33.163884 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.156672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36307 (local address 127.0.97.1:55891)
0619 01:11:33.157617 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:33.157634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:33.157842 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:33.158192 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:33.158196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:33.158220 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:33.158677 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:33.158688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.159662 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.159670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:33.161588 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:33.161600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.162766 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.162773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:33.162930 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:33.162952 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:11:33.163055 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 01:11:33.163182 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":100,"spinlock_wait_cycles":26880,"thread_start_us":436,"threads_started":1}
I20260619 01:11:33.164774 19466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c468d9bb79b49bdba7f43ef3ac1db4e" candidate_uuid: "1ccd94aeec0b46368c41a0c9284ed013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7a9178640e4524ae5c34739f77e198" is_pre_election: true
I20260619 01:11:33.164812 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c468d9bb79b49bdba7f43ef3ac1db4e" candidate_uuid: "1ccd94aeec0b46368c41a0c9284ed013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" is_pre_election: true
I20260619 01:11:33.165382 19466 raft_consensus.cc:2454] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccd94aeec0b46368c41a0c9284ed013 in term 0.
I20260619 01:11:33.165382 19596 raft_consensus.cc:2454] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ccd94aeec0b46368c41a0c9284ed013 in term 0.
I20260619 01:11:33.166132 19268 leader_election.cc:302] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ccd94aeec0b46368c41a0c9284ed013, 7d7a9178640e4524ae5c34739f77e198, 95b57fc7ee0647429deaa6d3c6449b6a; no voters:
I20260619 01:11:33.166533 19665 raft_consensus.cc:2781] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:11:33.166689 19665 raft_consensus.cc:482] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:11:33.166814 19665 raft_consensus.cc:3037] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:33.169109 19665 raft_consensus.cc:504] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7a9178640e4524ae5c34739f77e198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40801 } } peers { permanent_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36307 } } peers { permanent_uuid: "1ccd94aeec0b46368c41a0c9284ed013" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } }
W20260619 01:11:33.169281 19642 tablet.cc:2369] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:11:33.170435 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c468d9bb79b49bdba7f43ef3ac1db4e" candidate_uuid: "1ccd94aeec0b46368c41a0c9284ed013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95b57fc7ee0647429deaa6d3c6449b6a"
I20260619 01:11:33.170429 19466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c468d9bb79b49bdba7f43ef3ac1db4e" candidate_uuid: "1ccd94aeec0b46368c41a0c9284ed013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7a9178640e4524ae5c34739f77e198"
I20260619 01:11:33.170668 19596 raft_consensus.cc:3037] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:33.170668 19466 raft_consensus.cc:3037] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:33.171032 19665 leader_election.cc:288] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [CANDIDATE]: Term 1 election: Requested vote from peers 7d7a9178640e4524ae5c34739f77e198 (127.0.97.2:40801), 95b57fc7ee0647429deaa6d3c6449b6a (127.0.97.3:36307)
I20260619 01:11:33.173985 19466 raft_consensus.cc:2454] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccd94aeec0b46368c41a0c9284ed013 in term 1.
I20260619 01:11:33.173988 19596 raft_consensus.cc:2454] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ccd94aeec0b46368c41a0c9284ed013 in term 1.
I20260619 01:11:33.174430 19268 leader_election.cc:302] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ccd94aeec0b46368c41a0c9284ed013, 95b57fc7ee0647429deaa6d3c6449b6a; no voters:
I20260619 01:11:33.174832 19665 raft_consensus.cc:2781] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:33.175231 19665 raft_consensus.cc:686] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 1 LEADER]: Becoming Leader. State: Replica: 1ccd94aeec0b46368c41a0c9284ed013, State: Running, Role: LEADER
I20260619 01:11:33.175722 19665 consensus_queue.cc:237] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7a9178640e4524ae5c34739f77e198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40801 } } peers { permanent_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36307 } } peers { permanent_uuid: "1ccd94aeec0b46368c41a0c9284ed013" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } }
I20260619 01:11:33.180287 19193 catalog_manager.cc:5495] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ccd94aeec0b46368c41a0c9284ed013 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1ccd94aeec0b46368c41a0c9284ed013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7a9178640e4524ae5c34739f77e198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ccd94aeec0b46368c41a0c9284ed013" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44961 } health_report { overall_health: HEALTHY } } }
W20260619 01:11:33.203948 19512 tablet.cc:2369] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:11:33.218216 19381 tablet.cc:2369] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:11:33.229908 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.224282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.0.97.1:44961)
0619 01:11:33.224522 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:33.224526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:33.224545 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:11:33.224677 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:33.224681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:33.224770 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:33.224897 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:11:33.225415 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.226114 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.227596 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.228000 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.228097 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:33.228591 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:11:33.229106 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:11:33.229247 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:33.229528 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 01:11:33.229586 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 01:11:33.238533 19466 raft_consensus.cc:1261] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198 [term 1 FOLLOWER]: Refusing update from remote peer 1ccd94aeec0b46368c41a0c9284ed013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:33.238543 19596 raft_consensus.cc:1261] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a [term 1 FOLLOWER]: Refusing update from remote peer 1ccd94aeec0b46368c41a0c9284ed013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:33.239552 19670 consensus_queue.cc:1035] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d7a9178640e4524ae5c34739f77e198" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:33.239919 19665 consensus_queue.cc:1035] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95b57fc7ee0647429deaa6d3c6449b6a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:33.254963 19675 mvcc.cc:205] Tried to move back new op lower bound from 7298381796294959104 to 7298381796051632128. Current Snapshot: MvccSnapshot[applied={T|T < 7298381796294959104}]
I20260619 01:11:33.269413 19194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:33.269773 19194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:33.274056 19194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53a685c457cb411c8e23b0d50cfb362d: Sending DeleteTablet for 3 replicas of tablet 5c468d9bb79b49bdba7f43ef3ac1db4e
I20260619 01:11:33.275494 19446 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c468d9bb79b49bdba7f43ef3ac1db4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:33 UTC) from {username='slave'} at 127.0.0.1:43000
I20260619 01:11:33.275741 19315 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c468d9bb79b49bdba7f43ef3ac1db4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:33 UTC) from {username='slave'} at 127.0.0.1:44968
I20260619 01:11:33.276260 19576 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c468d9bb79b49bdba7f43ef3ac1db4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:33 UTC) from {username='slave'} at 127.0.0.1:42808
I20260619 01:11:33.277245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 19253
I20260619 01:11:33.288208 19682 tablet_replica.cc:330] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: stopping tablet replica
I20260619 01:11:33.288859 19682 raft_consensus.cc:2229] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:33.289620 19682 raft_consensus.cc:2258] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:33.290696 19684 tablet_replica.cc:330] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: stopping tablet replica
I20260619 01:11:33.291213 19684 raft_consensus.cc:2229] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:33.291839 19684 raft_consensus.cc:2258] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:33.294860 19684 ts_tablet_manager.cc:1905] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:33.297886 19682 ts_tablet_manager.cc:1905] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:33.304829 19684 ts_tablet_manager.cc:1918] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:33.305073 19684 log.cc:1196] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/5c468d9bb79b49bdba7f43ef3ac1db4e
I20260619 01:11:33.305234 19682 ts_tablet_manager.cc:1918] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:33.305825 19684 ts_tablet_manager.cc:1939] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 95b57fc7ee0647429deaa6d3c6449b6a: Deleting consensus metadata
I20260619 01:11:33.305886 19682 log.cc:1196] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/5c468d9bb79b49bdba7f43ef3ac1db4e
I20260619 01:11:33.306475 19683 tablet_replica.cc:330] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013: stopping tablet replica
I20260619 01:11:33.306494 19682 ts_tablet_manager.cc:1939] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 7d7a9178640e4524ae5c34739f77e198: Deleting consensus metadata
I20260619 01:11:33.306999 19683 raft_consensus.cc:2229] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:11:33.307976 19178 catalog_manager.cc:4841] TS 95b57fc7ee0647429deaa6d3c6449b6a (127.0.97.3:36307): tablet 5c468d9bb79b49bdba7f43ef3ac1db4e (table test-table [id=23aca0809ca04ff38406231d895f55b3]) successfully deleted
I20260619 01:11:33.308292 19683 raft_consensus.cc:2258] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:33.308301 19181 catalog_manager.cc:4841] TS 7d7a9178640e4524ae5c34739f77e198 (127.0.97.2:40801): tablet 5c468d9bb79b49bdba7f43ef3ac1db4e (table test-table [id=23aca0809ca04ff38406231d895f55b3]) successfully deleted
I20260619 01:11:33.310659 19683 ts_tablet_manager.cc:1905] T 5c468d9bb79b49bdba7f43ef3ac1db4e P 1ccd94aeec0b46368c41a0c9284ed013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:33.310945 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.282849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.0.97.1:44961)
0619 01:11:33.283105 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:33.283115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:33.291023 (+ 7908us) server_negotiation.cc:407] Connection header received
0619 01:11:33.291961 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:33.291968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:33.292111 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:33.292197 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:33.293034 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.294329 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.305651 (+ 11322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.306712 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.306881 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:33.309019 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:33.309034 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:33.309054 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:33.309973 (+ 919us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:33.310039 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:33.310067 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:33.310190 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:33.310384 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 01:11:33.310495 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 01:11:33.311787 19375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:33.418879 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 19253
I20260619 01:11:33.437836 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 19384
W20260619 01:11:33.437850 19178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44961 (error 108)
I20260619 01:11:33.439906 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.439452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44961 (local address 127.0.0.1:44986)
0619 01:11:33.439638 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:33.439758 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 01:11:33.440188 19178 catalog_manager.cc:4584] TS 1ccd94aeec0b46368c41a0c9284ed013 (127.0.97.1:44961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c468d9bb79b49bdba7f43ef3ac1db4e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44961: connect: Connection refused (error 111)
I20260619 01:11:33.445119 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.438812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.0.97.2:40801)
0619 01:11:33.439003 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:33.439011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:33.439033 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:11:33.439258 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:33.439264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:33.439373 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:33.439492 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:33.439883 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.440768 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.442140 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.442558 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.442692 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:33.443765 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:33.443774 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:33.443789 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:33.444350 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:33.444406 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:33.444425 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:33.444558 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:33.444692 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 01:11:33.444796 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:11:33.445966 19506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:33.493932 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.493479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44961 (local address 127.0.0.1:45002)
0619 01:11:33.493690 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:33.493779 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 01:11:33.495105 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.494755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44961 (local address 127.0.0.1:45008)
0619 01:11:33.494900 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:33.494973 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 01:11:33.536648 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 19384
I20260619 01:11:33.555140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 19514
I20260619 01:11:33.555995 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.555433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44961 (local address 127.0.0.1:45012)
0619 01:11:33.555679 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:33.555786 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 01:11:33.557124 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.556717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44961 (local address 127.0.0.1:45026)
0619 01:11:33.556879 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:33.556976 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 01:11:33.562785 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.556200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.0.97.3:36307)
0619 01:11:33.556415 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:33.556419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:33.556435 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:11:33.556511 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:33.556515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:33.556606 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:33.556743 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:33.557172 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.558084 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.559493 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:33.559914 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:33.560061 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:33.561222 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:33.561232 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:33.561248 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:33.561990 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:33.562046 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:33.562067 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:33.562195 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:33.562331 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 01:11:33.562457 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 01:11:33.563463 19636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:33.637765 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.637131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44961 (local address 127.0.0.1:45042)
0619 01:11:33.637409 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:33.637548 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 01:11:33.639043 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:33.638549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44961 (local address 127.0.0.1:45046)
0619 01:11:33.638774 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:33.638869 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260619 01:11:33.654873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 19514
I20260619 01:11:33.672173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 19163
I20260619 01:11:33.673453 19224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:33.785513 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 19163
2026-06-19T01:11:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2946 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:33 dist-test-slave-5p8w krb5kdc[19693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:33 dist-test-slave-5p8w krb5kdc[19693](info): set up 2 sockets
Jun 19 01:11:33 dist-test-slave-5p8w krb5kdc[19693](info): commencing operation
krb5kdc: starting...
W20260619 01:11:34.897670 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:35 dist-test-slave-5p8w krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:11:35.160559 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38867
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:35.508066 19706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:35.508394 19706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:35.508488 19706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:35.516055 19706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:35.516180 19706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:35.516228 19706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:35.516294 19706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:35.516340 19706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:35.526736 19706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:35.528234 19706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:35.529579 19706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:35.542273 19711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:35.542250 19712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:35.542209 19714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:35.543143 19706 server_base.cc:1034] running on GCE node
I20260619 01:11:35.544042 19706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:35.545632 19706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:35.546868 19706 hybrid_clock.cc:648] HybridClock initialized: now 1781831495546824 us; error 46 us; skew 500 ppm
I20260619 01:11:35.547245 19706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:35 dist-test-slave-5p8w krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:35.558059 19706 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:11:35.559612 19706 webserver.cc:460] Webserver started at http://127.0.97.62:35613/ using document root <none> and password file <none>
I20260619 01:11:35.560314 19706 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:35.560417 19706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:35.560676 19706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:35.563685 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "c236755aabc54a7fbc817dc3201ac64a"
format_stamp: "Formatted at 2026-06-19 01:11:35 on dist-test-slave-5p8w"
server_key: "bb84641ce746e405ce7ce8bcbece61e3"
server_key_iv: "9829dcfaa67c51b516cf85f487c29a63"
server_key_version: "encryptionkey@0"
I20260619 01:11:35.564395 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "c236755aabc54a7fbc817dc3201ac64a"
format_stamp: "Formatted at 2026-06-19 01:11:35 on dist-test-slave-5p8w"
server_key: "bb84641ce746e405ce7ce8bcbece61e3"
server_key_iv: "9829dcfaa67c51b516cf85f487c29a63"
server_key_version: "encryptionkey@0"
I20260619 01:11:35.569651 19706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 01:11:35.572913 19721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:35.574330 19706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:35.574517 19706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "c236755aabc54a7fbc817dc3201ac64a"
format_stamp: "Formatted at 2026-06-19 01:11:35 on dist-test-slave-5p8w"
server_key: "bb84641ce746e405ce7ce8bcbece61e3"
server_key_iv: "9829dcfaa67c51b516cf85f487c29a63"
server_key_version: "encryptionkey@0"
I20260619 01:11:35.574719 19706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:35.588032 19706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:35.592494 19706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:35.592783 19706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:35.611580 19706 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38867
I20260619 01:11:35.611613 19772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38867 every 8 connection(s)
I20260619 01:11:35.613058 19706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:35.616653 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 19706
I20260619 01:11:35.616869 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:35.617084 19773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:35.617436 388 external_mini_cluster.cc:1428] Setting key 91ae4e36cd6cce2fe456c29694e44bc9
I20260619 01:11:35.632122 19773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a: Bootstrap starting.
Jun 19 01:11:35 dist-test-slave-5p8w krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:35.636111 19773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:35.637516 19773 log.cc:824] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:35.641177 19773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a: No bootstrap required, opened a new log
I20260619 01:11:35.643263 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:35.619295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34050 (local address 127.0.97.62:38867)
0619 01:11:35.620138 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:35.620155 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:35.620210 (+ 55us) server_negotiation.cc:407] Connection header received
0619 01:11:35.622313 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:35.622343 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:35.622896 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:35.623387 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:35.624767 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:35.626840 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:35.628312 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:35.629520 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:35.629717 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:35.633947 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:35.633964 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:35.633988 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:35.634054 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:35.638127 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:35.639121 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:35.639133 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:35.639141 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:35.639337 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:35.639717 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:35.639721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:35.639724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:35.640458 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:35.640750 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:35.641182 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 01:11:35.641883 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":326,"threads_started":1}
I20260619 01:11:35.650480 19773 raft_consensus.cc:348] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c236755aabc54a7fbc817dc3201ac64a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38867 } }
I20260619 01:11:35.650866 19773 raft_consensus.cc:374] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:35.650980 19773 raft_consensus.cc:729] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c236755aabc54a7fbc817dc3201ac64a, State: Initialized, Role: FOLLOWER
I20260619 01:11:35.651626 19773 consensus_queue.cc:260] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c236755aabc54a7fbc817dc3201ac64a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38867 } }
I20260619 01:11:35.651851 19773 raft_consensus.cc:388] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:35.651950 19773 raft_consensus.cc:482] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:35.652097 19773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:35.654289 19773 raft_consensus.cc:504] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c236755aabc54a7fbc817dc3201ac64a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38867 } }
I20260619 01:11:35.654744 19773 leader_election.cc:302] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c236755aabc54a7fbc817dc3201ac64a; no voters:
I20260619 01:11:35.655264 19773 leader_election.cc:288] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:35.655416 19778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:35.656086 19778 raft_consensus.cc:686] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [term 1 LEADER]: Becoming Leader. State: Replica: c236755aabc54a7fbc817dc3201ac64a, State: Running, Role: LEADER
I20260619 01:11:35.656605 19778 consensus_queue.cc:237] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c236755aabc54a7fbc817dc3201ac64a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38867 } }
I20260619 01:11:35.657399 19773 sys_catalog.cc:564] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:35.659776 19780 sys_catalog.cc:455] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c236755aabc54a7fbc817dc3201ac64a. Latest consensus state: current_term: 1 leader_uuid: "c236755aabc54a7fbc817dc3201ac64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c236755aabc54a7fbc817dc3201ac64a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38867 } } }
I20260619 01:11:35.659804 19779 sys_catalog.cc:455] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c236755aabc54a7fbc817dc3201ac64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c236755aabc54a7fbc817dc3201ac64a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38867 } } }
I20260619 01:11:35.660269 19780 sys_catalog.cc:458] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:35.660274 19779 sys_catalog.cc:458] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:35.661100 19784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:35.668684 19784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:35.678326 19784 catalog_manager.cc:1347] Generated new cluster ID: bd87e7c407714d17b49eabc1cea8faf6
I20260619 01:11:35.678452 19784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:35.694492 19784 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:35.695852 19784 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:35.702832 19784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a: Generated new TSK 0
I20260619 01:11:35.703780 19784 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:11:35.772432 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38867
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:36.119092 19799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:36.119395 19799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:36.119503 19799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:36.127315 19799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:36.127453 19799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:36.127580 19799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:11:36.138114 19799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:36.139530 19799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:36.140863 19799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:36.155018 19804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:36.155040 19805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:36.154994 19807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:36.155647 19799 server_base.cc:1034] running on GCE node
I20260619 01:11:36.156324 19799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:36.157284 19799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:36.158577 19799 hybrid_clock.cc:648] HybridClock initialized: now 1781831496158526 us; error 45 us; skew 500 ppm
I20260619 01:11:36.158967 19799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:36 dist-test-slave-5p8w krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:36.162283 19799 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:11:36.163798 19799 webserver.cc:460] Webserver started at http://127.0.97.1:45169/ using document root <none> and password file <none>
I20260619 01:11:36.164544 19799 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:36.164695 19799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:36.165026 19799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:36.168171 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "56bc8603be244095a5fa1a71fc13341b"
format_stamp: "Formatted at 2026-06-19 01:11:36 on dist-test-slave-5p8w"
server_key: "e23ce51b06bb441824336d0eb215f409"
server_key_iv: "aedeb256a489e8b457b8358dc1b0df12"
server_key_version: "encryptionkey@0"
I20260619 01:11:36.168988 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "56bc8603be244095a5fa1a71fc13341b"
format_stamp: "Formatted at 2026-06-19 01:11:36 on dist-test-slave-5p8w"
server_key: "e23ce51b06bb441824336d0eb215f409"
server_key_iv: "aedeb256a489e8b457b8358dc1b0df12"
server_key_version: "encryptionkey@0"
I20260619 01:11:36.174525 19799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 01:11:36.178195 19814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:36.179708 19799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:11:36.179886 19799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "56bc8603be244095a5fa1a71fc13341b"
format_stamp: "Formatted at 2026-06-19 01:11:36 on dist-test-slave-5p8w"
server_key: "e23ce51b06bb441824336d0eb215f409"
server_key_iv: "aedeb256a489e8b457b8358dc1b0df12"
server_key_version: "encryptionkey@0"
I20260619 01:11:36.180095 19799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:36.203666 19799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:36.208271 19799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:36.208599 19799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:36.209733 19799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:36.211139 19799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:36.211221 19799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:36.211400 19799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:36.211464 19799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:36.244719 19799 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33963
I20260619 01:11:36.244822 19926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33963 every 8 connection(s)
I20260619 01:11:36.246146 19799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:11:36.248541 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 19799
I20260619 01:11:36.248719 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:11:36.249364 388 external_mini_cluster.cc:1428] Setting key c816cf312c916e320e194724983fde23
Jun 19 01:11:36 dist-test-slave-5p8w krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:36.267858 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:36.248995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43387 (local address 127.0.97.62:38867)
0619 01:11:36.249529 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:36.249536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:36.250696 (+ 1160us) server_negotiation.cc:407] Connection header received
0619 01:11:36.252863 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:36.252869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:36.252952 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:36.253086 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:36.254771 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:36.255487 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:36.257125 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:36.257687 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:36.257828 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:36.261863 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:36.261869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:36.261873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:36.261908 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:36.264579 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:36.265421 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:36.265426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:36.265428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:36.265557 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:36.266018 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:36.266029 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:36.266032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:36.266304 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:36.266467 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:36.267251 (+ 784us) server_negotiation.cc:299] Negotiation successful
0619 01:11:36.267487 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":268,"threads_started":1}
I20260619 01:11:36.268893 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:36.249644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38867 (local address 127.0.97.1:43387)
0619 01:11:36.250513 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:36.250580 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:36.252579 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:36.253408 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:36.253416 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:36.253972 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:36.254587 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:36.254606 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:36.255665 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:36.255670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:36.256961 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:36.256974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:36.257927 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:36.257936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:36.258212 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:36.259002 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:11:36.259043 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:36.261615 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:36.264743 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:36.264753 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:36.264773 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:36.265248 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:36.265676 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:36.265681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:36.265684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:36.265895 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:36.266448 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:36.266457 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:36.266827 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 01:11:36.267123 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 01:11:36.267689 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":323,"threads_started":1}
I20260619 01:11:36.270682 19927 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38867
I20260619 01:11:36.271060 19927 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:36.272091 19927 heartbeater.cc:511] Master 127.0.97.62:38867 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:11:36.274631 19738 ts_manager.cc:194] Registered new tserver with Master: 56bc8603be244095a5fa1a71fc13341b (127.0.97.1:33963)
I20260619 01:11:36.276160 19738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:11:36.309047 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38867
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:36.653430 19933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:36.653741 19933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:36.653925 19933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:36.661801 19933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:36.661939 19933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:36.662111 19933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:11:36.672339 19933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:36.673866 19933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:36.675355 19933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:36.689702 19938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:36.689719 19941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:36.689824 19939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:36.690447 19933 server_base.cc:1034] running on GCE node
I20260619 01:11:36.691092 19933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:36.692070 19933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:36.693341 19933 hybrid_clock.cc:648] HybridClock initialized: now 1781831496693264 us; error 55 us; skew 500 ppm
I20260619 01:11:36.693718 19933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:36 dist-test-slave-5p8w krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:36.696825 19933 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:11:36.698382 19933 webserver.cc:460] Webserver started at http://127.0.97.2:34253/ using document root <none> and password file <none>
I20260619 01:11:36.699092 19933 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:36.699235 19933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:36.699568 19933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:36.702667 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "3bd03c11411248988c1fbdc51d347b9b"
format_stamp: "Formatted at 2026-06-19 01:11:36 on dist-test-slave-5p8w"
server_key: "124903156a0999bf98d4378068df56b5"
server_key_iv: "aaea0d3b64bae5584f18acdc53fad475"
server_key_version: "encryptionkey@0"
I20260619 01:11:36.703522 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3bd03c11411248988c1fbdc51d347b9b"
format_stamp: "Formatted at 2026-06-19 01:11:36 on dist-test-slave-5p8w"
server_key: "124903156a0999bf98d4378068df56b5"
server_key_iv: "aaea0d3b64bae5584f18acdc53fad475"
server_key_version: "encryptionkey@0"
I20260619 01:11:36.708880 19933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 01:11:36.712812 19948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:36.714599 19933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:36.714794 19933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "3bd03c11411248988c1fbdc51d347b9b"
format_stamp: "Formatted at 2026-06-19 01:11:36 on dist-test-slave-5p8w"
server_key: "124903156a0999bf98d4378068df56b5"
server_key_iv: "aaea0d3b64bae5584f18acdc53fad475"
server_key_version: "encryptionkey@0"
I20260619 01:11:36.715029 19933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:36.740738 19933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:36.744777 19933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:36.745082 19933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:36.746130 19933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:36.747534 19933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:36.747637 19933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:36.747781 19933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:36.747853 19933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:36.782234 19933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43701
I20260619 01:11:36.782294 20060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43701 every 8 connection(s)
I20260619 01:11:36.783617 19933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:11:36.784710 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 19933
I20260619 01:11:36.784912 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:11:36.785476 388 external_mini_cluster.cc:1428] Setting key 3863293f4023b395b2fe1daa42f57c9f
Jun 19 01:11:36 dist-test-slave-5p8w krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:36.810467 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:36.786714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33863 (local address 127.0.97.62:38867)
0619 01:11:36.787271 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:36.787278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:36.788779 (+ 1501us) server_negotiation.cc:407] Connection header received
0619 01:11:36.791804 (+ 3025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:36.791810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:36.791905 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:36.792031 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:36.794319 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:36.795109 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:36.797894 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:36.798676 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:36.798846 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:36.802780 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:36.802789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:36.802797 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:36.802848 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:36.806673 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:36.807559 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:36.807568 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:36.807574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:36.807760 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:36.808239 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:36.808248 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:36.808254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:36.808665 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:36.808840 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:36.809639 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 01:11:36.809935 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":278,"threads_started":1}
I20260619 01:11:36.811187 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:36.787624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38867 (local address 127.0.97.2:33863)
0619 01:11:36.788561 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:36.788617 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:36.791452 (+ 2835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:36.792333 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:36.792347 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:36.793183 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:36.794109 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:36.794137 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:36.795277 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:36.795285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:36.797679 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:36.797701 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:36.798932 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:36.798936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:36.799164 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:36.799950 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:11:36.799975 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:36.802419 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:36.806862 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:36.806871 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:36.806887 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:36.807390 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:36.807881 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:36.807884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:36.807887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:36.808091 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:36.808794 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:36.808801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:36.809172 (+ 371us) client_negotiation.cc:769] Sending connection context
0619 01:11:36.809484 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 01:11:36.809945 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":347,"threads_started":1}
I20260619 01:11:36.812975 20061 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38867
I20260619 01:11:36.813393 20061 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:36.814433 20061 heartbeater.cc:511] Master 127.0.97.62:38867 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:11:36.816284 19738 ts_manager.cc:194] Registered new tserver with Master: 3bd03c11411248988c1fbdc51d347b9b (127.0.97.2:43701)
I20260619 01:11:36.817221 19738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:11:36.848451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38867
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:37.187682 20067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:37.188019 20067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:37.188194 20067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:37.195680 20067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:37.195866 20067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:37.196064 20067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:11:37.205876 20067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:37.207208 20067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:37.208511 20067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:37.222462 20072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:37.222366 20073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:37.222406 20075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:37.223821 20067 server_base.cc:1034] running on GCE node
I20260619 01:11:37.224486 20067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:37.225623 20067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:37.226845 20067 hybrid_clock.cc:648] HybridClock initialized: now 1781831497226773 us; error 68 us; skew 500 ppm
I20260619 01:11:37.227205 20067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:37 dist-test-slave-5p8w krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:37.230576 20067 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:11:37.232132 20067 webserver.cc:460] Webserver started at http://127.0.97.3:41699/ using document root <none> and password file <none>
I20260619 01:11:37.232836 20067 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:37.232936 20067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:37.233206 20067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:37.236363 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "c2e3cb412c244957a1f785ab640dcbf9"
format_stamp: "Formatted at 2026-06-19 01:11:37 on dist-test-slave-5p8w"
server_key: "c56f18bb1de637a78750730f2c0a474d"
server_key_iv: "f345ba986c3eb2ff3bf230851d395c89"
server_key_version: "encryptionkey@0"
I20260619 01:11:37.237186 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c2e3cb412c244957a1f785ab640dcbf9"
format_stamp: "Formatted at 2026-06-19 01:11:37 on dist-test-slave-5p8w"
server_key: "c56f18bb1de637a78750730f2c0a474d"
server_key_iv: "f345ba986c3eb2ff3bf230851d395c89"
server_key_version: "encryptionkey@0"
I20260619 01:11:37.242940 20067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 01:11:37.247529 20082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:37.249359 20067 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 01:11:37.249549 20067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "c2e3cb412c244957a1f785ab640dcbf9"
format_stamp: "Formatted at 2026-06-19 01:11:37 on dist-test-slave-5p8w"
server_key: "c56f18bb1de637a78750730f2c0a474d"
server_key_iv: "f345ba986c3eb2ff3bf230851d395c89"
server_key_version: "encryptionkey@0"
I20260619 01:11:37.249794 20067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:37.268553 20067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:37.273373 20067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:37.273633 20067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:37.274611 20067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:37.276010 20067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:37.276082 20067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:37.276170 20067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:37.276212 20067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:37.279757 19927 heartbeater.cc:503] Master 127.0.97.62:38867 was elected leader, sending a full tablet report...
I20260619 01:11:37.311444 20067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44453
I20260619 01:11:37.311498 20194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44453 every 8 connection(s)
I20260619 01:11:37.312901 20067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:11:37.313511 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 20067
I20260619 01:11:37.313704 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:11:37.314293 388 external_mini_cluster.cc:1428] Setting key ef45329137cc1d8dad7a592506206d67
Jun 19 01:11:37 dist-test-slave-5p8w krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:11:37.334733 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.315890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56541 (local address 127.0.97.62:38867)
0619 01:11:37.316385 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:37.316391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:37.317646 (+ 1255us) server_negotiation.cc:407] Connection header received
0619 01:11:37.319649 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:37.319653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:37.319746 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:37.319843 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:37.321440 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.322261 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.323900 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.324322 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.324427 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.328266 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:37.328275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:37.328283 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:37.328334 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:37.331515 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:37.332445 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:37.332449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:37.332452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:37.332608 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:37.333000 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:37.333003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:37.333005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:37.333281 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:37.333474 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:37.334247 (+ 773us) server_negotiation.cc:299] Negotiation successful
0619 01:11:37.334439 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":229,"threads_started":1}
I20260619 01:11:37.335714 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.316575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38867 (local address 127.0.97.3:56541)
0619 01:11:37.317449 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:37.317510 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:37.319354 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:37.320080 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:37.320089 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:37.320649 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:37.321228 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.321246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.322413 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.322416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.323678 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.323691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.324642 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.324648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.324855 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.325604 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:11:37.325633 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:37.327938 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:37.331657 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:37.331665 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:37.331680 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:37.332159 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:37.332707 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:11:37.332710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:11:37.332713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:11:37.332896 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:11:37.333467 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:37.333474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:37.333860 (+ 386us) client_negotiation.cc:769] Sending connection context
0619 01:11:37.334126 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 01:11:37.334555 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":309,"threads_started":1}
I20260619 01:11:37.337280 20195 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38867
I20260619 01:11:37.337661 20195 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:37.338430 20195 heartbeater.cc:511] Master 127.0.97.62:38867 requested a full tablet report, sending...
I20260619 01:11:37.340013 19738 ts_manager.cc:194] Registered new tserver with Master: c2e3cb412c244957a1f785ab640dcbf9 (127.0.97.3:44453)
I20260619 01:11:37.340965 19738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56541
I20260619 01:11:37.342633 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:11:37.360958 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.347176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.0.97.62:38867)
0619 01:11:37.347403 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:37.347412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:37.347779 (+ 367us) server_negotiation.cc:407] Connection header received
0619 01:11:37.347993 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:37.348001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:37.348089 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:37.348196 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:37.349764 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.350530 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.351864 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.352300 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.352442 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.355199 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:37.355204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:37.355208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:37.355245 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:37.357932 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:37.358717 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:37.358723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:37.358727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:37.358863 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:37.359421 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:37.359425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:37.359428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:37.359768 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:37.360063 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:37.360400 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 01:11:37.360640 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:11:37.365217 19738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:11:37.372570 19738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:11:37.423420 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.385549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43701 (local address 127.0.0.1:33676)
0619 01:11:37.410397 (+ 24848us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:37.410461 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:37.410785 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:37.414422 (+ 3637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:37.414426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:37.414452 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:37.414865 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.414876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.417208 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.417216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.419333 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.419350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.421461 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.421471 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.422351 (+ 880us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.422432 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 01:11:37.422584 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 01:11:37.422821 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24464,"mutex_wait_us":72,"thread_start_us":27285,"threads_started":1}
I20260619 01:11:37.424235 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.385760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44453 (local address 127.0.0.1:43156)
0619 01:11:37.412133 (+ 26373us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:37.412164 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:37.412362 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:37.413708 (+ 1346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:37.413745 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:37.413783 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:37.414198 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.414210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.416286 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.416291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.420981 (+ 4690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.420998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.423320 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.423327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.423537 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.423587 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 01:11:37.423670 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 01:11:37.423817 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26144,"mutex_wait_us":43,"thread_start_us":27723,"threads_started":1}
I20260619 01:11:37.424643 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.385815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.0.97.2:43701)
0619 01:11:37.413841 (+ 28026us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:37.413849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:37.413875 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:11:37.413971 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:37.413979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:37.414225 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:37.414443 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:37.415108 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.417048 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.419543 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.421188 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.421368 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.421606 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:37.424082 (+ 2476us) server_negotiation.cc:299] Negotiation successful
0619 01:11:37.424305 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27852,"thread_start_us":27908,"threads_started":1}
I20260619 01:11:37.425097 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.386458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.0.97.3:44453)
0619 01:11:37.411669 (+ 25211us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:37.411681 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:37.412541 (+ 860us) server_negotiation.cc:407] Connection header received
0619 01:11:37.412686 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:37.412698 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:37.413038 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:37.413260 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:37.414430 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.416105 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.421233 (+ 5128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.423091 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.423233 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.423479 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:37.424517 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0619 01:11:37.424735 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24971,"thread_start_us":24759,"threads_started":1}
I20260619 01:11:37.425397 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.386190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33963 (local address 127.0.0.1:37786)
0619 01:11:37.413187 (+ 26997us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:37.413212 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:37.414159 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:37.414878 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:37.414882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:37.414914 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:37.415267 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.415278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.417470 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.417475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.422334 (+ 4859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.422350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.424503 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.424509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.424725 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.424760 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 01:11:37.424850 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 01:11:37.424991 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26823,"mutex_wait_us":2360,"thread_start_us":25526,"threads_started":1}
I20260619 01:11:37.426383 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.386781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.0.97.1:33963)
0619 01:11:37.409325 (+ 22544us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:37.409621 (+ 296us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:37.413374 (+ 3753us) server_negotiation.cc:407] Connection header received
0619 01:11:37.414388 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:37.414398 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:37.414668 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:37.414879 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:37.415434 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.417254 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.422576 (+ 5322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.424029 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.424115 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.424275 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:37.424916 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 01:11:37.425114 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22227,"thread_start_us":22537,"threads_started":1}
I20260619 01:11:37.429033 19996 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2bc0d0b4d340d19b98398f4666bb77 (DEFAULT_TABLE table=test-table [id=738328fd44a34537af1841c32c1b4f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:37.429033 20130 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2bc0d0b4d340d19b98398f4666bb77 (DEFAULT_TABLE table=test-table [id=738328fd44a34537af1841c32c1b4f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:37.429487 19862 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2bc0d0b4d340d19b98398f4666bb77 (DEFAULT_TABLE table=test-table [id=738328fd44a34537af1841c32c1b4f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:37.430706 19996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2bc0d0b4d340d19b98398f4666bb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:37.430706 20130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2bc0d0b4d340d19b98398f4666bb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:37.430747 19862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2bc0d0b4d340d19b98398f4666bb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:37.443492 20214 tablet_bootstrap.cc:492] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: Bootstrap starting.
I20260619 01:11:37.448134 20214 tablet_bootstrap.cc:654] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:37.449146 20216 tablet_bootstrap.cc:492] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: Bootstrap starting.
I20260619 01:11:37.449712 20214 log.cc:824] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:37.450105 20215 tablet_bootstrap.cc:492] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: Bootstrap starting.
I20260619 01:11:37.453792 20214 tablet_bootstrap.cc:492] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: No bootstrap required, opened a new log
I20260619 01:11:37.454113 20214 ts_tablet_manager.cc:1397] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260619 01:11:37.454129 20215 tablet_bootstrap.cc:654] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:37.455348 20216 tablet_bootstrap.cc:654] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:37.455485 20215 log.cc:824] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:37.456599 20216 log.cc:824] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:37.459053 20215 tablet_bootstrap.cc:492] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: No bootstrap required, opened a new log
I20260619 01:11:37.459379 20215 ts_tablet_manager.cc:1397] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 01:11:37.460053 20216 tablet_bootstrap.cc:492] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: No bootstrap required, opened a new log
I20260619 01:11:37.460328 20216 ts_tablet_manager.cc:1397] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260619 01:11:37.463557 20214 raft_consensus.cc:348] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd03c11411248988c1fbdc51d347b9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43701 } } peers { permanent_uuid: "c2e3cb412c244957a1f785ab640dcbf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44453 } } peers { permanent_uuid: "56bc8603be244095a5fa1a71fc13341b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33963 } }
I20260619 01:11:37.463972 20214 raft_consensus.cc:374] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:37.464143 20214 raft_consensus.cc:729] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bc8603be244095a5fa1a71fc13341b, State: Initialized, Role: FOLLOWER
I20260619 01:11:37.464779 20214 consensus_queue.cc:260] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd03c11411248988c1fbdc51d347b9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43701 } } peers { permanent_uuid: "c2e3cb412c244957a1f785ab640dcbf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44453 } } peers { permanent_uuid: "56bc8603be244095a5fa1a71fc13341b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33963 } }
I20260619 01:11:37.466771 20214 ts_tablet_manager.cc:1428] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260619 01:11:37.470577 20215 raft_consensus.cc:348] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd03c11411248988c1fbdc51d347b9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43701 } } peers { permanent_uuid: "c2e3cb412c244957a1f785ab640dcbf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44453 } } peers { permanent_uuid: "56bc8603be244095a5fa1a71fc13341b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33963 } }
I20260619 01:11:37.471061 20215 raft_consensus.cc:374] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:37.471171 20215 raft_consensus.cc:729] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e3cb412c244957a1f785ab640dcbf9, State: Initialized, Role: FOLLOWER
I20260619 01:11:37.471830 20215 consensus_queue.cc:260] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd03c11411248988c1fbdc51d347b9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43701 } } peers { permanent_uuid: "c2e3cb412c244957a1f785ab640dcbf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44453 } } peers { permanent_uuid: "56bc8603be244095a5fa1a71fc13341b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33963 } }
I20260619 01:11:37.472126 20216 raft_consensus.cc:348] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd03c11411248988c1fbdc51d347b9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43701 } } peers { permanent_uuid: "c2e3cb412c244957a1f785ab640dcbf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44453 } } peers { permanent_uuid: "56bc8603be244095a5fa1a71fc13341b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33963 } }
I20260619 01:11:37.472656 20216 raft_consensus.cc:374] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:37.472795 20216 raft_consensus.cc:729] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd03c11411248988c1fbdc51d347b9b, State: Initialized, Role: FOLLOWER
I20260619 01:11:37.472999 20195 heartbeater.cc:503] Master 127.0.97.62:38867 was elected leader, sending a full tablet report...
I20260619 01:11:37.473650 20216 consensus_queue.cc:260] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd03c11411248988c1fbdc51d347b9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43701 } } peers { permanent_uuid: "c2e3cb412c244957a1f785ab640dcbf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44453 } } peers { permanent_uuid: "56bc8603be244095a5fa1a71fc13341b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33963 } }
I20260619 01:11:37.473856 20215 ts_tablet_manager.cc:1428] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260619 01:11:37.474771 20061 heartbeater.cc:503] Master 127.0.97.62:38867 was elected leader, sending a full tablet report...
I20260619 01:11:37.476511 20216 ts_tablet_manager.cc:1428] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
W20260619 01:11:37.498680 19928 tablet.cc:2369] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:11:37.535959 20062 tablet.cc:2369] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:11:37.536059 20222 raft_consensus.cc:482] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:11:37.536399 20222 raft_consensus.cc:504] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd03c11411248988c1fbdc51d347b9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43701 } } peers { permanent_uuid: "c2e3cb412c244957a1f785ab640dcbf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44453 } } peers { permanent_uuid: "56bc8603be244095a5fa1a71fc13341b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33963 } }
I20260619 01:11:37.538445 20222 leader_election.cc:288] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2e3cb412c244957a1f785ab640dcbf9 (127.0.97.3:44453), 56bc8603be244095a5fa1a71fc13341b (127.0.97.1:33963)
I20260619 01:11:37.547551 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.538786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33963 (local address 127.0.97.2:45667)
0619 01:11:37.540197 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:37.540215 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:37.540404 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:37.540779 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:37.540783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:37.540807 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:37.541221 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.541235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.542581 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.542589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.544950 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.544967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.546734 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.546741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.546936 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.546963 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 01:11:37.547028 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 01:11:37.547128 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1293,"thread_start_us":1393,"threads_started":1}
I20260619 01:11:37.548334 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.539126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54363 (local address 127.0.97.3:44453)
0619 01:11:37.539630 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:37.539639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:37.540075 (+ 436us) server_negotiation.cc:407] Connection header received
0619 01:11:37.540858 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:37.540865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:37.540959 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:37.541034 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:37.541972 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.543174 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.545814 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.547145 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.547278 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.547381 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:37.547951 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 01:11:37.548080 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331}
I20260619 01:11:37.548789 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.539900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45667 (local address 127.0.97.1:33963)
0619 01:11:37.540116 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:37.540124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:37.540295 (+ 171us) server_negotiation.cc:407] Connection header received
0619 01:11:37.540583 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:37.540589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:37.540661 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:37.540761 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:37.541440 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.542421 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.545149 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.546469 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.548245 (+ 1776us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.548345 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:37.548415 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 01:11:37.548557 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 01:11:37.549088 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.538682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44453 (local address 127.0.97.2:54363)
0619 01:11:37.539926 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:37.539962 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:37.540222 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:37.541242 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:37.541247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:37.541276 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:37.541765 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.541778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.543352 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.543358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.545630 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:37.545643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.547554 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.547560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:37.547733 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.547772 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 01:11:37.548595 (+ 823us) client_negotiation.cc:240] Negotiation successful
0619 01:11:37.548740 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"thread_start_us":880,"threads_started":1}
I20260619 01:11:37.549893 19882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2bc0d0b4d340d19b98398f4666bb77" candidate_uuid: "3bd03c11411248988c1fbdc51d347b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bc8603be244095a5fa1a71fc13341b" is_pre_election: true
I20260619 01:11:37.550081 20150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2bc0d0b4d340d19b98398f4666bb77" candidate_uuid: "3bd03c11411248988c1fbdc51d347b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e3cb412c244957a1f785ab640dcbf9" is_pre_election: true
I20260619 01:11:37.550501 20150 raft_consensus.cc:2454] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd03c11411248988c1fbdc51d347b9b in term 0.
I20260619 01:11:37.550496 19882 raft_consensus.cc:2454] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd03c11411248988c1fbdc51d347b9b in term 0.
I20260619 01:11:37.551208 19949 leader_election.cc:302] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd03c11411248988c1fbdc51d347b9b, 56bc8603be244095a5fa1a71fc13341b; no voters:
I20260619 01:11:37.551702 20222 raft_consensus.cc:2781] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:11:37.551872 20222 raft_consensus.cc:482] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:11:37.551985 20222 raft_consensus.cc:3037] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:37.554365 20222 raft_consensus.cc:504] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd03c11411248988c1fbdc51d347b9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43701 } } peers { permanent_uuid: "c2e3cb412c244957a1f785ab640dcbf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44453 } } peers { permanent_uuid: "56bc8603be244095a5fa1a71fc13341b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33963 } }
I20260619 01:11:37.555015 20222 leader_election.cc:288] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [CANDIDATE]: Term 1 election: Requested vote from peers c2e3cb412c244957a1f785ab640dcbf9 (127.0.97.3:44453), 56bc8603be244095a5fa1a71fc13341b (127.0.97.1:33963)
I20260619 01:11:37.555629 20150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2bc0d0b4d340d19b98398f4666bb77" candidate_uuid: "3bd03c11411248988c1fbdc51d347b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e3cb412c244957a1f785ab640dcbf9"
I20260619 01:11:37.555631 19882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2bc0d0b4d340d19b98398f4666bb77" candidate_uuid: "3bd03c11411248988c1fbdc51d347b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56bc8603be244095a5fa1a71fc13341b"
I20260619 01:11:37.555884 20150 raft_consensus.cc:3037] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:37.555876 19882 raft_consensus.cc:3037] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:37.559262 20150 raft_consensus.cc:2454] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd03c11411248988c1fbdc51d347b9b in term 1.
I20260619 01:11:37.559262 19882 raft_consensus.cc:2454] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd03c11411248988c1fbdc51d347b9b in term 1.
I20260619 01:11:37.559819 19952 leader_election.cc:302] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd03c11411248988c1fbdc51d347b9b, c2e3cb412c244957a1f785ab640dcbf9; no voters:
I20260619 01:11:37.560176 20222 raft_consensus.cc:2781] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:37.560643 20222 raft_consensus.cc:686] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 1 LEADER]: Becoming Leader. State: Replica: 3bd03c11411248988c1fbdc51d347b9b, State: Running, Role: LEADER
I20260619 01:11:37.561147 20222 consensus_queue.cc:237] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd03c11411248988c1fbdc51d347b9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43701 } } peers { permanent_uuid: "c2e3cb412c244957a1f785ab640dcbf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44453 } } peers { permanent_uuid: "56bc8603be244095a5fa1a71fc13341b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33963 } }
W20260619 01:11:37.564628 20196 tablet.cc:2369] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:11:37.566347 19737 catalog_manager.cc:5495] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bd03c11411248988c1fbdc51d347b9b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3bd03c11411248988c1fbdc51d347b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd03c11411248988c1fbdc51d347b9b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2e3cb412c244957a1f785ab640dcbf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56bc8603be244095a5fa1a71fc13341b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33963 } health_report { overall_health: UNKNOWN } } }
I20260619 01:11:37.590158 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.584940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.0.97.2:43701)
0619 01:11:37.585152 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:37.585156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:37.585175 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:11:37.585421 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:37.585425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:37.585502 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:37.585633 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:11:37.585951 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.586647 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.588027 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.588410 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.588504 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.588948 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:11:37.589557 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:11:37.589667 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:37.589765 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 01:11:37.589828 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:11:37.598481 20150 raft_consensus.cc:1261] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9 [term 1 FOLLOWER]: Refusing update from remote peer 3bd03c11411248988c1fbdc51d347b9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:37.598481 19882 raft_consensus.cc:1261] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b [term 1 FOLLOWER]: Refusing update from remote peer 3bd03c11411248988c1fbdc51d347b9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:37.599701 20225 consensus_queue.cc:1035] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "56bc8603be244095a5fa1a71fc13341b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:37.600066 20222 consensus_queue.cc:1035] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e3cb412c244957a1f785ab640dcbf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:37.618395 20229 mvcc.cc:205] Tried to move back new op lower bound from 7298381814151856128 to 7298381814015336448. Current Snapshot: MvccSnapshot[applied={T|T < 7298381814151856128}]
I20260619 01:11:37.645988 19738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:37.646342 19738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:37.650385 19738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c236755aabc54a7fbc817dc3201ac64a: Sending DeleteTablet for 3 replicas of tablet 8a2bc0d0b4d340d19b98398f4666bb77
I20260619 01:11:37.651690 20130 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2bc0d0b4d340d19b98398f4666bb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43156
I20260619 01:11:37.651685 19996 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2bc0d0b4d340d19b98398f4666bb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33676
I20260619 01:11:37.651942 19862 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2bc0d0b4d340d19b98398f4666bb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37786
I20260619 01:11:37.654565 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 19799
I20260619 01:11:37.667811 20239 tablet_replica.cc:330] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: stopping tablet replica
I20260619 01:11:37.668447 20239 raft_consensus.cc:2229] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:11:37.668789 20238 tablet_replica.cc:330] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: stopping tablet replica
I20260619 01:11:37.669329 20238 raft_consensus.cc:2229] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:37.669404 20239 raft_consensus.cc:2258] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:37.670228 20238 raft_consensus.cc:2258] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:37.671015 20240 tablet_replica.cc:330] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: stopping tablet replica
I20260619 01:11:37.671466 20240 raft_consensus.cc:2229] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:37.672021 20240 raft_consensus.cc:2258] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:37.674139 20238 ts_tablet_manager.cc:1905] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:37.674546 20239 ts_tablet_manager.cc:1905] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:37.675382 20240 ts_tablet_manager.cc:1905] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 01:11:37 dist-test-slave-5p8w krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:11:37.681638 20238 ts_tablet_manager.cc:1918] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:37.681921 20238 log.cc:1196] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/8a2bc0d0b4d340d19b98398f4666bb77
I20260619 01:11:37.682565 20238 ts_tablet_manager.cc:1939] T 8a2bc0d0b4d340d19b98398f4666bb77 P c2e3cb412c244957a1f785ab640dcbf9: Deleting consensus metadata
I20260619 01:11:37.684609 19725 catalog_manager.cc:4841] TS c2e3cb412c244957a1f785ab640dcbf9 (127.0.97.3:44453): tablet 8a2bc0d0b4d340d19b98398f4666bb77 (table test-table [id=738328fd44a34537af1841c32c1b4f52]) successfully deleted
I20260619 01:11:37.687554 20240 ts_tablet_manager.cc:1918] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:37.687747 20240 log.cc:1196] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/8a2bc0d0b4d340d19b98398f4666bb77
I20260619 01:11:37.688355 20240 ts_tablet_manager.cc:1939] T 8a2bc0d0b4d340d19b98398f4666bb77 P 56bc8603be244095a5fa1a71fc13341b: Deleting consensus metadata
I20260619 01:11:37.690092 20239 ts_tablet_manager.cc:1918] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:37.690346 20239 log.cc:1196] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/8a2bc0d0b4d340d19b98398f4666bb77
I20260619 01:11:37.690460 19722 catalog_manager.cc:4841] TS 56bc8603be244095a5fa1a71fc13341b (127.0.97.1:33963): tablet 8a2bc0d0b4d340d19b98398f4666bb77 (table test-table [id=738328fd44a34537af1841c32c1b4f52]) successfully deleted
I20260619 01:11:37.690949 20239 ts_tablet_manager.cc:1939] T 8a2bc0d0b4d340d19b98398f4666bb77 P 3bd03c11411248988c1fbdc51d347b9b: Deleting consensus metadata
I20260619 01:11:37.691570 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.655978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.0.97.1:33963)
0619 01:11:37.656212 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:37.656219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:37.670304 (+ 14085us) server_negotiation.cc:407] Connection header received
0619 01:11:37.670534 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:37.670540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:37.670674 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:37.670995 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:37.672904 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.674139 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.676123 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.676794 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.676905 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.682957 (+ 6052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:37.682976 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:37.682997 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:37.683059 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:37.687496 (+ 4437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:37.688499 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:37.688511 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:37.688515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:37.688694 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:37.689206 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:37.689212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:37.689215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:37.689811 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:37.690115 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:37.690911 (+ 796us) server_negotiation.cc:299] Negotiation successful
0619 01:11:37.691163 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 01:11:37.692589 19723 catalog_manager.cc:4841] TS 3bd03c11411248988c1fbdc51d347b9b (127.0.97.2:43701): tablet 8a2bc0d0b4d340d19b98398f4666bb77 (table test-table [id=738328fd44a34537af1841c32c1b4f52]) successfully deleted
I20260619 01:11:37.693070 19922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:37.807094 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 19799
I20260619 01:11:37.825421 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 19933
Jun 19 01:11:37 dist-test-slave-5p8w krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:11:37.843302 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.826344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.0.97.2:43701)
0619 01:11:37.826554 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:37.826560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:37.826584 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:11:37.826657 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:37.826662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:37.826750 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:37.826860 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:37.828147 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.829386 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.830675 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.831372 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.831507 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.836076 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:37.836098 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:37.836122 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:37.836172 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:37.840193 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:37.840932 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:37.840942 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:37.840946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:37.841152 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:37.841584 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:37.841589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:37.841593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:37.842065 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:37.842397 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:37.842621 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 01:11:37.842857 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:11:37.844170 20056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:37.959553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 19933
I20260619 01:11:37.979151 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 20067
Jun 19 01:11:37 dist-test-slave-5p8w krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:11:37.994928 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:37.979939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.0.97.3:44453)
0619 01:11:37.980103 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:37.980108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:37.980128 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:11:37.980306 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:37.980310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:37.980376 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:37.980465 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:37.981730 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.982440 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.983795 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:37.984188 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:37.984284 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:37.989152 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:37.989169 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:37.989186 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:37.989223 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:37.991961 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:37.992738 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:37.992746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:37.992748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:37.992894 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:37.993350 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:37.993355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:37.993358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:37.993729 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:37.994044 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:37.994351 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 01:11:37.994576 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 01:11:37.995605 20190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:38.086619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 20067
I20260619 01:11:38.103467 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 19706
I20260619 01:11:38.104681 19768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:38.219686 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 19706
2026-06-19T01:11:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4435 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T01:11:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:38Z Disabled control of system clock
I20260619 01:11:38.299629 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34417
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:11:38.643188 20249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:38.643574 20249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:38.643726 20249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:38.651259 20249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:38.651439 20249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:38.651540 20249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:38.651643 20249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:38.651719 20249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:38.661134 20249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:38.662642 20249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:38.664654 20249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:38.677403 20254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:38.677388 20255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:38.677379 20257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:38.678388 20249 server_base.cc:1034] running on GCE node
I20260619 01:11:38.679296 20249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:38.680785 20249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:38.682080 20249 hybrid_clock.cc:648] HybridClock initialized: now 1781831498682012 us; error 67 us; skew 500 ppm
I20260619 01:11:38.682479 20249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:38.684911 20249 webserver.cc:460] Webserver started at http://127.0.97.62:39447/ using document root <none> and password file <none>
I20260619 01:11:38.685732 20249 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:38.685871 20249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:38.686208 20249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:38.689203 20249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "ac457ac5ad7d44d2abaceaf5cdb7c5dc"
format_stamp: "Formatted at 2026-06-19 01:11:38 on dist-test-slave-5p8w"
server_key: "6ab59cb2e340aa19c2fc42b72417feaf"
server_key_iv: "842ce25e0af755cc46e29e630543304c"
server_key_version: "encryptionkey@0"
I20260619 01:11:38.690125 20249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "ac457ac5ad7d44d2abaceaf5cdb7c5dc"
format_stamp: "Formatted at 2026-06-19 01:11:38 on dist-test-slave-5p8w"
server_key: "6ab59cb2e340aa19c2fc42b72417feaf"
server_key_iv: "842ce25e0af755cc46e29e630543304c"
server_key_version: "encryptionkey@0"
I20260619 01:11:38.695420 20249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:11:38.698944 20263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:38.700359 20249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:11:38.700553 20249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "ac457ac5ad7d44d2abaceaf5cdb7c5dc"
format_stamp: "Formatted at 2026-06-19 01:11:38 on dist-test-slave-5p8w"
server_key: "6ab59cb2e340aa19c2fc42b72417feaf"
server_key_iv: "842ce25e0af755cc46e29e630543304c"
server_key_version: "encryptionkey@0"
I20260619 01:11:38.700794 20249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:38.729992 20249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:38.730898 20249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:38.731207 20249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:38.750391 20249 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34417
I20260619 01:11:38.750411 20314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34417 every 8 connection(s)
I20260619 01:11:38.751861 20249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:38.754819 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 20249
I20260619 01:11:38.755014 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:38.755541 388 external_mini_cluster.cc:1428] Setting key 409fb698c96a8033e8d6689d0e3dd485
I20260619 01:11:38.756531 20315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:38.769202 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:38.757502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.0.97.62:34417)
0619 01:11:38.758516 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:38.758535 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:38.758577 (+ 42us) server_negotiation.cc:407] Connection header received
0619 01:11:38.760147 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:38.760169 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:38.760677 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:38.761096 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:38.761739 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:38.763076 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:38.764480 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:38.765209 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:38.765448 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:38.766703 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:38.766713 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:38.766758 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:38.767400 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:38.767489 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:38.767518 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:38.767656 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:38.767889 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 01:11:38.768226 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":327,"threads_started":1}
I20260619 01:11:38.769663 20315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc: Bootstrap starting.
I20260619 01:11:38.773073 20315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:38.774221 20315 log.cc:824] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:38.777186 20315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc: No bootstrap required, opened a new log
I20260619 01:11:38.785245 20315 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac457ac5ad7d44d2abaceaf5cdb7c5dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34417 } }
I20260619 01:11:38.785763 20315 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:38.785874 20315 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac457ac5ad7d44d2abaceaf5cdb7c5dc, State: Initialized, Role: FOLLOWER
I20260619 01:11:38.786532 20315 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac457ac5ad7d44d2abaceaf5cdb7c5dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34417 } }
I20260619 01:11:38.786782 20315 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:38.786926 20315 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:38.787124 20315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:38.789906 20315 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac457ac5ad7d44d2abaceaf5cdb7c5dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34417 } }
I20260619 01:11:38.790462 20315 leader_election.cc:302] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac457ac5ad7d44d2abaceaf5cdb7c5dc; no voters:
I20260619 01:11:38.791062 20315 leader_election.cc:288] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:38.791193 20320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:38.792009 20320 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [term 1 LEADER]: Becoming Leader. State: Replica: ac457ac5ad7d44d2abaceaf5cdb7c5dc, State: Running, Role: LEADER
I20260619 01:11:38.792608 20320 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac457ac5ad7d44d2abaceaf5cdb7c5dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34417 } }
I20260619 01:11:38.793442 20315 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:38.795601 20322 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac457ac5ad7d44d2abaceaf5cdb7c5dc. Latest consensus state: current_term: 1 leader_uuid: "ac457ac5ad7d44d2abaceaf5cdb7c5dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac457ac5ad7d44d2abaceaf5cdb7c5dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34417 } } }
I20260619 01:11:38.796097 20322 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:38.796631 20321 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac457ac5ad7d44d2abaceaf5cdb7c5dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac457ac5ad7d44d2abaceaf5cdb7c5dc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34417 } } }
I20260619 01:11:38.796981 20321 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:38.798799 20329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:38.803218 20329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:38.812492 20329 catalog_manager.cc:1347] Generated new cluster ID: bbb8f9d13d654e09b5530174a7c47e7a
I20260619 01:11:38.812633 20329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:38.841845 20329 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:38.843802 20329 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:38.851536 20329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc: Generated new TSK 0
I20260619 01:11:38.852550 20329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:38.866789 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34417
--builtin_ntp_servers=127.0.97.20:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:11:39.217702 20339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:39.218024 20339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:39.218112 20339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:39.225630 20339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:39.225750 20339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:39.225862 20339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:11:39.235586 20339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:39.237012 20339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:39.238783 20339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:39.252574 20344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:39.252542 20347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:39.252496 20345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:39.253690 20339 server_base.cc:1034] running on GCE node
I20260619 01:11:39.254453 20339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:39.255491 20339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:39.256750 20339 hybrid_clock.cc:648] HybridClock initialized: now 1781831499256700 us; error 44 us; skew 500 ppm
I20260619 01:11:39.257117 20339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:39.259636 20339 webserver.cc:460] Webserver started at http://127.0.97.1:35371/ using document root <none> and password file <none>
I20260619 01:11:39.260416 20339 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:39.260555 20339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:39.260911 20339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:39.264248 20339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/data/instance:
uuid: "098ef57413564e44a49c71a5830c50e7"
format_stamp: "Formatted at 2026-06-19 01:11:39 on dist-test-slave-5p8w"
server_key: "adc7ea5506ee25ba5db54905ef3ba273"
server_key_iv: "6004df4e274f95d31329dc32d2774308"
server_key_version: "encryptionkey@0"
I20260619 01:11:39.265122 20339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/wal/instance:
uuid: "098ef57413564e44a49c71a5830c50e7"
format_stamp: "Formatted at 2026-06-19 01:11:39 on dist-test-slave-5p8w"
server_key: "adc7ea5506ee25ba5db54905ef3ba273"
server_key_iv: "6004df4e274f95d31329dc32d2774308"
server_key_version: "encryptionkey@0"
I20260619 01:11:39.270823 20339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 01:11:39.274804 20353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:39.276501 20339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:11:39.276722 20339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/wal
uuid: "098ef57413564e44a49c71a5830c50e7"
format_stamp: "Formatted at 2026-06-19 01:11:39 on dist-test-slave-5p8w"
server_key: "adc7ea5506ee25ba5db54905ef3ba273"
server_key_iv: "6004df4e274f95d31329dc32d2774308"
server_key_version: "encryptionkey@0"
I20260619 01:11:39.277012 20339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:39.294502 20339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:39.295454 20339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:39.295758 20339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:39.296873 20339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:39.298727 20339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:39.298885 20339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:39.299023 20339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:39.299114 20339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:39.333226 20339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43587
I20260619 01:11:39.333243 20465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43587 every 8 connection(s)
I20260619 01:11:39.334674 20339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:11:39.343649 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 20339
I20260619 01:11:39.343840 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:11:39.344303 388 external_mini_cluster.cc:1428] Setting key 87edc07f2cc40f90779f632fc5118859
I20260619 01:11:39.346993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34417
--builtin_ntp_servers=127.0.97.20:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:11:39.348014 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:39.337462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40933 (local address 127.0.97.62:34417)
0619 01:11:39.337915 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:39.337921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:39.339260 (+ 1339us) server_negotiation.cc:407] Connection header received
0619 01:11:39.341064 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:39.341069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:39.341140 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:39.341241 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:39.342451 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:39.343186 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:39.344949 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:39.345470 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:39.345628 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:39.347112 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:39.347117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:39.347120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:39.347160 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:39.347208 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:39.347225 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:39.347314 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:39.347564 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 01:11:39.347669 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":200,"threads_started":1}
I20260619 01:11:39.349150 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:39.338158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34417 (local address 127.0.97.1:40933)
0619 01:11:39.339091 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:39.339129 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:39.340773 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:39.341539 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:39.341548 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:39.341646 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:39.342276 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:39.342292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:39.343344 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:39.343348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:39.344802 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:39.344814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:39.345736 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:39.345743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:39.346010 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:39.346757 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:11:39.346760 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:39.346805 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:11:39.346888 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:39.347313 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:39.347329 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:39.347340 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 01:11:39.347537 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 01:11:39.347981 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":282,"threads_started":1}
I20260619 01:11:39.351059 20466 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34417
I20260619 01:11:39.351377 20466 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:39.352200 20466 heartbeater.cc:511] Master 127.0.97.62:34417 requested a full tablet report, sending...
I20260619 01:11:39.354686 20280 ts_manager.cc:194] Registered new tserver with Master: 098ef57413564e44a49c71a5830c50e7 (127.0.97.1:43587)
I20260619 01:11:39.356309 20280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:40933
W20260619 01:11:39.683849 20470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:39.684154 20470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:39.684243 20470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:39.691592 20470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:39.691716 20470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:39.691826 20470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:11:39.701450 20470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:39.702756 20470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:39.704610 20470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:39.719062 20475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:39.719081 20476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:39.719110 20478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:39.719920 20470 server_base.cc:1034] running on GCE node
I20260619 01:11:39.720543 20470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:39.721603 20470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:39.722954 20470 hybrid_clock.cc:648] HybridClock initialized: now 1781831499722776 us; error 35 us; skew 500 ppm
I20260619 01:11:39.723349 20470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:39.725955 20470 webserver.cc:460] Webserver started at http://127.0.97.2:33971/ using document root <none> and password file <none>
I20260619 01:11:39.726745 20470 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:39.726867 20470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:39.727226 20470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:39.730337 20470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/data/instance:
uuid: "f5014542e16a4e738cc844d6e8bd081c"
format_stamp: "Formatted at 2026-06-19 01:11:39 on dist-test-slave-5p8w"
server_key: "d6d1ed4bedbdd5b35339629750457873"
server_key_iv: "4e674da551709c26bddd2116fc4a1228"
server_key_version: "encryptionkey@0"
I20260619 01:11:39.731182 20470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f5014542e16a4e738cc844d6e8bd081c"
format_stamp: "Formatted at 2026-06-19 01:11:39 on dist-test-slave-5p8w"
server_key: "d6d1ed4bedbdd5b35339629750457873"
server_key_iv: "4e674da551709c26bddd2116fc4a1228"
server_key_version: "encryptionkey@0"
I20260619 01:11:39.736981 20470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:11:39.740868 20484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:39.742453 20470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:39.742672 20470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/wal
uuid: "f5014542e16a4e738cc844d6e8bd081c"
format_stamp: "Formatted at 2026-06-19 01:11:39 on dist-test-slave-5p8w"
server_key: "d6d1ed4bedbdd5b35339629750457873"
server_key_iv: "4e674da551709c26bddd2116fc4a1228"
server_key_version: "encryptionkey@0"
I20260619 01:11:39.742920 20470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:39.767154 20470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:39.768110 20470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:39.768425 20470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:39.769637 20470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:39.771238 20470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:39.771363 20470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:39.771507 20470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:39.771600 20470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:39.805891 20470 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38055
I20260619 01:11:39.805984 20596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38055 every 8 connection(s)
I20260619 01:11:39.807264 20470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:11:39.814962 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 20470
I20260619 01:11:39.815164 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:11:39.815697 388 external_mini_cluster.cc:1428] Setting key fcfbc761c797ff99791348bd7a6f5259
I20260619 01:11:39.818866 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34417
--builtin_ntp_servers=127.0.97.20:38897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:11:39.821696 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:39.810187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34337 (local address 127.0.97.62:34417)
0619 01:11:39.810418 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:39.810423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:39.811847 (+ 1424us) server_negotiation.cc:407] Connection header received
0619 01:11:39.813824 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:39.813829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:39.813927 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:39.814042 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:39.815500 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:39.816310 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:39.818482 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:39.819179 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:39.819318 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:39.820750 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:39.820759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:39.820767 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:39.820794 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:39.820838 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:39.820857 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:39.820956 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:39.821172 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 01:11:39.821284 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 01:11:39.822708 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:39.810733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34417 (local address 127.0.97.2:34337)
0619 01:11:39.811674 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:39.811711 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:39.813535 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:39.814294 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:39.814303 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:39.814407 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:39.815288 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:39.815317 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:39.816485 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:39.816490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:39.818317 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:39.818330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:39.819413 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:39.819418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:39.819637 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:39.820390 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:11:39.820392 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:39.820436 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:11:39.820529 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:39.820957 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:39.820965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:39.820970 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 01:11:39.821123 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 01:11:39.821529 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":288,"threads_started":1}
I20260619 01:11:39.824303 20597 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34417
I20260619 01:11:39.824651 20597 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:39.825503 20597 heartbeater.cc:511] Master 127.0.97.62:34417 requested a full tablet report, sending...
I20260619 01:11:39.827219 20280 ts_manager.cc:194] Registered new tserver with Master: f5014542e16a4e738cc844d6e8bd081c (127.0.97.2:38055)
I20260619 01:11:39.828272 20280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:34337
W20260619 01:11:40.163947 20600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:40.164273 20600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:40.164484 20600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:40.172434 20600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:40.172572 20600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:40.172768 20600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:11:40.182502 20600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:40.183957 20600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:40.186199 20600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:40.201581 20605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:40.201527 20606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:40.201613 20608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:40.202349 20600 server_base.cc:1034] running on GCE node
I20260619 01:11:40.203063 20600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:40.204183 20600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:40.205399 20600 hybrid_clock.cc:648] HybridClock initialized: now 1781831500205343 us; error 41 us; skew 500 ppm
I20260619 01:11:40.205758 20600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:11:40.208113 20600 webserver.cc:460] Webserver started at http://127.0.97.3:36721/ using document root <none> and password file <none>
I20260619 01:11:40.208865 20600 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:40.208998 20600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:40.209414 20600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:40.212544 20600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/data/instance:
uuid: "593ea93ec7744155b8c4b737e1abc168"
format_stamp: "Formatted at 2026-06-19 01:11:40 on dist-test-slave-5p8w"
server_key: "f833b3bb88122d3f0a2c2f258aa1d084"
server_key_iv: "8b2e9f467a030bef1507d11f526a5f87"
server_key_version: "encryptionkey@0"
I20260619 01:11:40.213514 20600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/wal/instance:
uuid: "593ea93ec7744155b8c4b737e1abc168"
format_stamp: "Formatted at 2026-06-19 01:11:40 on dist-test-slave-5p8w"
server_key: "f833b3bb88122d3f0a2c2f258aa1d084"
server_key_iv: "8b2e9f467a030bef1507d11f526a5f87"
server_key_version: "encryptionkey@0"
I20260619 01:11:40.218905 20600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:11:40.222541 20614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:40.223905 20600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:11:40.224097 20600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/wal
uuid: "593ea93ec7744155b8c4b737e1abc168"
format_stamp: "Formatted at 2026-06-19 01:11:40 on dist-test-slave-5p8w"
server_key: "f833b3bb88122d3f0a2c2f258aa1d084"
server_key_iv: "8b2e9f467a030bef1507d11f526a5f87"
server_key_version: "encryptionkey@0"
I20260619 01:11:40.224346 20600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:40.243640 20600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:40.244553 20600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:40.244864 20600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:40.245916 20600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:11:40.247609 20600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:11:40.247727 20600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:40.247859 20600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:11:40.247956 20600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:40.283985 20600 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45687
I20260619 01:11:40.284045 20726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45687 every 8 connection(s)
I20260619 01:11:40.285503 20600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:11:40.295279 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 20600
I20260619 01:11:40.295451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:11:40.295985 388 external_mini_cluster.cc:1428] Setting key d2199991a23807152006050fa08bfaae
I20260619 01:11:40.299583 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.288230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59073 (local address 127.0.97.62:34417)
0619 01:11:40.288471 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:40.288478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:40.290274 (+ 1796us) server_negotiation.cc:407] Connection header received
0619 01:11:40.292107 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:40.292112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:40.292180 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:40.292271 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:40.293355 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.294229 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.296327 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.296943 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.297086 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.298762 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:40.298767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:40.298770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:40.298788 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:40.298842 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:40.298861 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:40.298946 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:40.299236 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 01:11:40.299341 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:11:40.300699 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.288905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34417 (local address 127.0.97.3:59073)
0619 01:11:40.290111 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:40.290154 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:40.291814 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:40.292469 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:40.292478 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:40.292588 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:11:40.293148 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.293164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.294361 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.294365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.296195 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.296209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.297175 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.297182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.297506 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.298332 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:11:40.298336 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:11:40.298397 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:11:40.298509 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:11:40.298971 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:11:40.298982 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:11:40.298991 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 01:11:40.299206 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 01:11:40.299536 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":425,"threads_started":1}
I20260619 01:11:40.301986 20727 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34417
I20260619 01:11:40.302299 20727 heartbeater.cc:464] Registering TS with master...
I20260619 01:11:40.303009 20727 heartbeater.cc:511] Master 127.0.97.62:34417 requested a full tablet report, sending...
I20260619 01:11:40.304356 20280 ts_manager.cc:194] Registered new tserver with Master: 593ea93ec7744155b8c4b737e1abc168 (127.0.97.3:45687)
I20260619 01:11:40.305167 20280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:59073
I20260619 01:11:40.311362 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:11:40.321038 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.315425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.0.97.62:34417)
0619 01:11:40.315628 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:40.315633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:40.316093 (+ 460us) server_negotiation.cc:407] Connection header received
0619 01:11:40.316239 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:40.316243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:40.316313 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:40.316395 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:40.316819 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.317601 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.318935 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.319321 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.319413 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.320477 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:40.320481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:40.320484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:40.320513 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:40.320543 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:40.320555 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:40.320621 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:40.320737 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 01:11:40.320811 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 01:11:40.324975 20280 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:11:40.332070 20280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:11:40.356621 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.345593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.0.97.2:38055)
0619 01:11:40.347389 (+ 1796us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:40.347398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:40.347422 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:11:40.347519 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:40.347529 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:40.347812 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:40.348755 (+ 943us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:40.349001 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.350701 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.353863 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.355374 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.355522 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.355765 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:40.356068 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 01:11:40.356240 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1634,"thread_start_us":1532,"threads_started":1}
I20260619 01:11:40.356634 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.345426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38055 (local address 127.0.0.1:59858)
0619 01:11:40.346334 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:40.346370 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:40.346656 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:40.348065 (+ 1409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:40.348072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:40.348103 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:40.348530 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.348560 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.350868 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.350875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.353635 (+ 2760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.353660 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.355589 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.355596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.355794 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.355855 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 01:11:40.356024 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 01:11:40.356240 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":32,"thread_start_us":540,"threads_started":1}
I20260619 01:11:40.356849 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.346237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.0.97.1:43587)
0619 01:11:40.346977 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:40.346988 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:40.347361 (+ 373us) server_negotiation.cc:407] Connection header received
0619 01:11:40.347737 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:40.347746 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:40.347940 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:40.348119 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:40.349063 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.350706 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.353978 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.355402 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.355573 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.355833 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:40.356312 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 01:11:40.356480 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":424,"threads_started":1}
I20260619 01:11:40.357637 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.345623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.0.1:35460)
0619 01:11:40.347249 (+ 1626us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:40.347267 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:40.347469 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:40.348425 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:40.348435 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:40.348463 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:40.348862 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.348876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.350888 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.350894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.353740 (+ 2846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.353754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.355913 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.355919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.356092 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.356125 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 01:11:40.357118 (+ 993us) client_negotiation.cc:240] Negotiation successful
0619 01:11:40.357254 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1484,"mutex_wait_us":1,"thread_start_us":1341,"threads_started":1}
I20260619 01:11:40.359663 20466 heartbeater.cc:503] Master 127.0.97.62:34417 was elected leader, sending a full tablet report...
I20260619 01:11:40.360627 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.347792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.0.97.3:45687)
0619 01:11:40.351403 (+ 3611us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:40.351413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:40.352146 (+ 733us) server_negotiation.cc:407] Connection header received
0619 01:11:40.352234 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:40.352239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:40.352425 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:40.352610 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:40.353205 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.354667 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.357143 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.358601 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.358751 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.359030 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:40.360084 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0619 01:11:40.360264 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3434,"thread_start_us":3279,"threads_started":1}
I20260619 01:11:40.359882 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.347404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45687 (local address 127.0.0.1:43842)
0619 01:11:40.351840 (+ 4436us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:40.351859 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:40.351982 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:40.352605 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:40.352609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:40.352633 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:40.353027 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.353036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.354837 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.354844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.356917 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.356933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.359150 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.359155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.359351 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.359383 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:11:40.359473 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 01:11:40.359602 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4301,"thread_start_us":4139,"threads_started":1}
I20260619 01:11:40.361231 20532 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffc5226be654a60aafe9b4ab9739a67 (DEFAULT_TABLE table=test-table [id=60b6f4c6217946dc96ede5a60b6e1cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:40.361236 20401 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffc5226be654a60aafe9b4ab9739a67 (DEFAULT_TABLE table=test-table [id=60b6f4c6217946dc96ede5a60b6e1cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:40.362778 20532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffc5226be654a60aafe9b4ab9739a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:40.362778 20401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffc5226be654a60aafe9b4ab9739a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:40.364183 20662 tablet_service.cc:1467] Processing CreateTablet for tablet 8ffc5226be654a60aafe9b4ab9739a67 (DEFAULT_TABLE table=test-table [id=60b6f4c6217946dc96ede5a60b6e1cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:11:40.365722 20662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ffc5226be654a60aafe9b4ab9739a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:40.377537 20745 tablet_bootstrap.cc:492] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168: Bootstrap starting.
I20260619 01:11:40.381150 20745 tablet_bootstrap.cc:654] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:40.381989 20747 tablet_bootstrap.cc:492] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7: Bootstrap starting.
I20260619 01:11:40.382859 20745 log.cc:824] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:40.383891 20746 tablet_bootstrap.cc:492] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c: Bootstrap starting.
I20260619 01:11:40.385876 20747 tablet_bootstrap.cc:654] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:40.386152 20745 tablet_bootstrap.cc:492] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168: No bootstrap required, opened a new log
I20260619 01:11:40.386432 20745 ts_tablet_manager.cc:1397] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260619 01:11:40.387162 20747 log.cc:824] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:40.387552 20746 tablet_bootstrap.cc:654] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:40.389046 20746 log.cc:824] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:40.391893 20747 tablet_bootstrap.cc:492] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7: No bootstrap required, opened a new log
I20260619 01:11:40.392230 20747 ts_tablet_manager.cc:1397] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260619 01:11:40.392488 20746 tablet_bootstrap.cc:492] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c: No bootstrap required, opened a new log
I20260619 01:11:40.392810 20746 ts_tablet_manager.cc:1397] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260619 01:11:40.397253 20745 raft_consensus.cc:348] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5014542e16a4e738cc844d6e8bd081c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38055 } } peers { permanent_uuid: "098ef57413564e44a49c71a5830c50e7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "593ea93ec7744155b8c4b737e1abc168" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45687 } }
I20260619 01:11:40.397785 20745 raft_consensus.cc:374] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:40.397890 20745 raft_consensus.cc:729] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593ea93ec7744155b8c4b737e1abc168, State: Initialized, Role: FOLLOWER
I20260619 01:11:40.398617 20745 consensus_queue.cc:260] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5014542e16a4e738cc844d6e8bd081c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38055 } } peers { permanent_uuid: "098ef57413564e44a49c71a5830c50e7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "593ea93ec7744155b8c4b737e1abc168" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45687 } }
I20260619 01:11:40.399866 20727 heartbeater.cc:503] Master 127.0.97.62:34417 was elected leader, sending a full tablet report...
I20260619 01:11:40.400573 20745 ts_tablet_manager.cc:1428] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260619 01:11:40.400898 20747 raft_consensus.cc:348] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5014542e16a4e738cc844d6e8bd081c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38055 } } peers { permanent_uuid: "098ef57413564e44a49c71a5830c50e7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "593ea93ec7744155b8c4b737e1abc168" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45687 } }
I20260619 01:11:40.401408 20747 raft_consensus.cc:374] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:40.401527 20747 raft_consensus.cc:729] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098ef57413564e44a49c71a5830c50e7, State: Initialized, Role: FOLLOWER
I20260619 01:11:40.402153 20747 consensus_queue.cc:260] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5014542e16a4e738cc844d6e8bd081c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38055 } } peers { permanent_uuid: "098ef57413564e44a49c71a5830c50e7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "593ea93ec7744155b8c4b737e1abc168" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45687 } }
I20260619 01:11:40.403389 20746 raft_consensus.cc:348] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5014542e16a4e738cc844d6e8bd081c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38055 } } peers { permanent_uuid: "098ef57413564e44a49c71a5830c50e7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "593ea93ec7744155b8c4b737e1abc168" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45687 } }
I20260619 01:11:40.403883 20746 raft_consensus.cc:374] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:40.404001 20746 raft_consensus.cc:729] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5014542e16a4e738cc844d6e8bd081c, State: Initialized, Role: FOLLOWER
I20260619 01:11:40.404809 20746 consensus_queue.cc:260] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5014542e16a4e738cc844d6e8bd081c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38055 } } peers { permanent_uuid: "098ef57413564e44a49c71a5830c50e7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "593ea93ec7744155b8c4b737e1abc168" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45687 } }
I20260619 01:11:40.405921 20597 heartbeater.cc:503] Master 127.0.97.62:34417 was elected leader, sending a full tablet report...
I20260619 01:11:40.406759 20746 ts_tablet_manager.cc:1428] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260619 01:11:40.409153 20747 ts_tablet_manager.cc:1428] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7: Time spent starting tablet: real 0.017s user 0.016s sys 0.002s
I20260619 01:11:40.440145 20751 raft_consensus.cc:482] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:11:40.440522 20751 raft_consensus.cc:504] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5014542e16a4e738cc844d6e8bd081c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38055 } } peers { permanent_uuid: "098ef57413564e44a49c71a5830c50e7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "593ea93ec7744155b8c4b737e1abc168" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45687 } }
I20260619 01:11:40.442394 20751 leader_election.cc:288] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5014542e16a4e738cc844d6e8bd081c (127.0.97.2:38055), 098ef57413564e44a49c71a5830c50e7 (127.0.97.1:43587)
I20260619 01:11:40.449327 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.442859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56843 (local address 127.0.97.1:43587)
0619 01:11:40.443092 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:40.443099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:40.443282 (+ 183us) server_negotiation.cc:407] Connection header received
0619 01:11:40.443520 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:40.443525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:40.443636 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:40.443760 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:40.444301 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.445202 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.447231 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.448317 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.448429 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.448549 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:40.448892 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 01:11:40.449039 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 01:11:40.449419 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.442751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43587 (local address 127.0.97.3:56843)
0619 01:11:40.443133 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:40.443161 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:40.443345 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:40.443891 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:40.443896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:40.443925 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:40.444124 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.444139 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.445358 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.445363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.447038 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.447052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.448482 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.448490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.448703 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.448749 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 01:11:40.448855 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 01:11:40.449008 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":88}
I20260619 01:11:40.449625 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.442783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38055 (local address 127.0.97.3:49309)
0619 01:11:40.443642 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 01:11:40.443660 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:11:40.443790 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:11:40.444190 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:11:40.444193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:11:40.444236 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:11:40.444582 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.444597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.445707 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.445711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.447374 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:11:40.447389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.448629 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.448641 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:11:40.448817 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.448851 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 01:11:40.449027 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 01:11:40.449171 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":42,"thread_start_us":490,"threads_started":1}
I20260619 01:11:40.449998 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.442927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49309 (local address 127.0.97.2:38055)
0619 01:11:40.443103 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:40.443109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:40.443762 (+ 653us) server_negotiation.cc:407] Connection header received
0619 01:11:40.443955 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:40.443962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:40.444069 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:40.444169 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:11:40.444756 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.445568 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.447563 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.448451 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.448597 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.448692 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:40.449060 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 01:11:40.449200 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:11:40.450488 20421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffc5226be654a60aafe9b4ab9739a67" candidate_uuid: "593ea93ec7744155b8c4b737e1abc168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098ef57413564e44a49c71a5830c50e7" is_pre_election: true
I20260619 01:11:40.450915 20552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffc5226be654a60aafe9b4ab9739a67" candidate_uuid: "593ea93ec7744155b8c4b737e1abc168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5014542e16a4e738cc844d6e8bd081c" is_pre_election: true
I20260619 01:11:40.451068 20421 raft_consensus.cc:2454] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593ea93ec7744155b8c4b737e1abc168 in term 0.
I20260619 01:11:40.451393 20552 raft_consensus.cc:2454] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593ea93ec7744155b8c4b737e1abc168 in term 0.
I20260619 01:11:40.451807 20615 leader_election.cc:302] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098ef57413564e44a49c71a5830c50e7, 593ea93ec7744155b8c4b737e1abc168; no voters:
I20260619 01:11:40.452270 20751 raft_consensus.cc:2781] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:11:40.452428 20751 raft_consensus.cc:482] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:11:40.452558 20751 raft_consensus.cc:3037] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:40.455041 20751 raft_consensus.cc:504] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5014542e16a4e738cc844d6e8bd081c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38055 } } peers { permanent_uuid: "098ef57413564e44a49c71a5830c50e7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "593ea93ec7744155b8c4b737e1abc168" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45687 } }
I20260619 01:11:40.455646 20751 leader_election.cc:288] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [CANDIDATE]: Term 1 election: Requested vote from peers f5014542e16a4e738cc844d6e8bd081c (127.0.97.2:38055), 098ef57413564e44a49c71a5830c50e7 (127.0.97.1:43587)
I20260619 01:11:40.456135 20552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffc5226be654a60aafe9b4ab9739a67" candidate_uuid: "593ea93ec7744155b8c4b737e1abc168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5014542e16a4e738cc844d6e8bd081c"
I20260619 01:11:40.456151 20421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ffc5226be654a60aafe9b4ab9739a67" candidate_uuid: "593ea93ec7744155b8c4b737e1abc168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098ef57413564e44a49c71a5830c50e7"
I20260619 01:11:40.456348 20552 raft_consensus.cc:3037] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:40.456362 20421 raft_consensus.cc:3037] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:40.458678 20421 raft_consensus.cc:2454] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593ea93ec7744155b8c4b737e1abc168 in term 1.
I20260619 01:11:40.458688 20552 raft_consensus.cc:2454] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593ea93ec7744155b8c4b737e1abc168 in term 1.
I20260619 01:11:40.459112 20615 leader_election.cc:302] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098ef57413564e44a49c71a5830c50e7, 593ea93ec7744155b8c4b737e1abc168; no voters:
I20260619 01:11:40.459429 20751 raft_consensus.cc:2781] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:40.459913 20751 raft_consensus.cc:686] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 1 LEADER]: Becoming Leader. State: Replica: 593ea93ec7744155b8c4b737e1abc168, State: Running, Role: LEADER
I20260619 01:11:40.460418 20751 consensus_queue.cc:237] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5014542e16a4e738cc844d6e8bd081c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38055 } } peers { permanent_uuid: "098ef57413564e44a49c71a5830c50e7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } } peers { permanent_uuid: "593ea93ec7744155b8c4b737e1abc168" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45687 } }
I20260619 01:11:40.465591 20280 catalog_manager.cc:5495] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 593ea93ec7744155b8c4b737e1abc168 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "593ea93ec7744155b8c4b737e1abc168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5014542e16a4e738cc844d6e8bd081c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "098ef57413564e44a49c71a5830c50e7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593ea93ec7744155b8c4b737e1abc168" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45687 } health_report { overall_health: HEALTHY } } }
I20260619 01:11:40.490437 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.484526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43844 (local address 127.0.97.3:45687)
0619 01:11:40.484716 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:40.484723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:40.484826 (+ 103us) server_negotiation.cc:407] Connection header received
0619 01:11:40.485089 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:40.485093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:40.485185 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:40.485361 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:11:40.485764 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.486605 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.488126 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.488563 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.488705 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.489202 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:11:40.489836 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:11:40.490008 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:40.490079 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 01:11:40.490165 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:11:40.499213 20552 raft_consensus.cc:1261] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c [term 1 FOLLOWER]: Refusing update from remote peer 593ea93ec7744155b8c4b737e1abc168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:40.499212 20421 raft_consensus.cc:1261] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7 [term 1 FOLLOWER]: Refusing update from remote peer 593ea93ec7744155b8c4b737e1abc168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:11:40.500262 20751 consensus_queue.cc:1035] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098ef57413564e44a49c71a5830c50e7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:40.500574 20755 consensus_queue.cc:1035] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5014542e16a4e738cc844d6e8bd081c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:11:40.529816 20279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:40.530277 20279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:11:40.536484 20279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac457ac5ad7d44d2abaceaf5cdb7c5dc: Sending DeleteTablet for 3 replicas of tablet 8ffc5226be654a60aafe9b4ab9739a67
I20260619 01:11:40.538163 20532 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffc5226be654a60aafe9b4ab9739a67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:40 UTC) from {username='slave'} at 127.0.0.1:59858
I20260619 01:11:40.541879 20401 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffc5226be654a60aafe9b4ab9739a67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:40 UTC) from {username='slave'} at 127.0.0.1:35460
I20260619 01:11:40.542300 20662 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ffc5226be654a60aafe9b4ab9739a67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:11:40 UTC) from {username='slave'} at 127.0.0.1:43842
I20260619 01:11:40.551915 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 20339
I20260619 01:11:40.566303 20769 tablet_replica.cc:330] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7: stopping tablet replica
I20260619 01:11:40.577721 20770 tablet_replica.cc:330] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168: stopping tablet replica
I20260619 01:11:40.581332 20770 raft_consensus.cc:2229] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:11:40.582562 20768 tablet_replica.cc:330] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c: stopping tablet replica
I20260619 01:11:40.583154 20768 raft_consensus.cc:2229] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:40.583806 20768 raft_consensus.cc:2258] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:40.587617 20768 ts_tablet_manager.cc:1905] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:40.591460 20770 raft_consensus.cc:2258] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:40.591554 20769 raft_consensus.cc:2229] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:11:40.592530 20769 raft_consensus.cc:2258] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:11:40.594553 20770 ts_tablet_manager.cc:1905] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:40.598224 20769 ts_tablet_manager.cc:1905] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:11:40.601855 20768 ts_tablet_manager.cc:1918] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:40.602099 20768 log.cc:1196] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-1/wal/wals/8ffc5226be654a60aafe9b4ab9739a67
I20260619 01:11:40.602422 20770 ts_tablet_manager.cc:1918] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:40.602607 20770 log.cc:1196] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-2/wal/wals/8ffc5226be654a60aafe9b4ab9739a67
I20260619 01:11:40.603124 20770 ts_tablet_manager.cc:1939] T 8ffc5226be654a60aafe9b4ab9739a67 P 593ea93ec7744155b8c4b737e1abc168: Deleting consensus metadata
I20260619 01:11:40.605080 20264 catalog_manager.cc:4841] TS 593ea93ec7744155b8c4b737e1abc168 (127.0.97.3:45687): tablet 8ffc5226be654a60aafe9b4ab9739a67 (table test-table [id=60b6f4c6217946dc96ede5a60b6e1cf7]) successfully deleted
I20260619 01:11:40.605346 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.553663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.0.97.1:43587)
0619 01:11:40.553912 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:40.553921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:40.576772 (+ 22851us) server_negotiation.cc:407] Connection header received
0619 01:11:40.577176 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:40.577187 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:40.577478 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:40.577584 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:40.578492 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.580085 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.593102 (+ 13017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.593862 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.593971 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.603010 (+ 9039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:40.603027 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:40.603057 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:40.603915 (+ 858us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:40.603982 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:40.604014 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:40.604155 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:40.604768 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 01:11:40.604899 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 01:11:40.605635 20769 ts_tablet_manager.cc:1918] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:11:40.605808 20769 log.cc:1196] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7: Deleting WAL directory at /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781831171286417-388-0/minicluster-data/ts-0/wal/wals/8ffc5226be654a60aafe9b4ab9739a67
I20260619 01:11:40.606503 20768 ts_tablet_manager.cc:1939] T 8ffc5226be654a60aafe9b4ab9739a67 P f5014542e16a4e738cc844d6e8bd081c: Deleting consensus metadata
I20260619 01:11:40.606635 20461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:40.608815 20267 catalog_manager.cc:4841] TS f5014542e16a4e738cc844d6e8bd081c (127.0.97.2:38055): tablet 8ffc5226be654a60aafe9b4ab9739a67 (table test-table [id=60b6f4c6217946dc96ede5a60b6e1cf7]) successfully deleted
I20260619 01:11:40.608947 20769 ts_tablet_manager.cc:1939] T 8ffc5226be654a60aafe9b4ab9739a67 P 098ef57413564e44a49c71a5830c50e7: Deleting consensus metadata
I20260619 01:11:40.708057 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 20339
I20260619 01:11:40.708273 20264 catalog_manager.cc:4841] TS 098ef57413564e44a49c71a5830c50e7 (127.0.97.1:43587): tablet 8ffc5226be654a60aafe9b4ab9739a67 (table test-table [id=60b6f4c6217946dc96ede5a60b6e1cf7]) successfully deleted
I20260619 01:11:40.726854 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 20470
I20260619 01:11:40.734961 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.727702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.0.97.2:38055)
0619 01:11:40.727937 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:40.727944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:40.728085 (+ 141us) server_negotiation.cc:407] Connection header received
0619 01:11:40.728282 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:40.728286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:40.728413 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:40.728531 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:40.729075 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.729916 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.731715 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.732138 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.732235 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.733538 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:40.733548 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:40.733563 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:40.734182 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:40.734234 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:40.734254 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:40.734349 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:40.734512 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 01:11:40.734609 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:11:40.735693 20592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:40.827059 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 20470
I20260619 01:11:40.844648 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 20600
I20260619 01:11:40.853003 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:40.845557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.0.97.3:45687)
0619 01:11:40.845766 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:40.845777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:40.845803 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:11:40.845880 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:40.845885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:40.845991 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:40.846134 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:40.846508 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.847746 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.849090 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:40.849935 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:40.850095 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:40.851230 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:40.851243 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:40.851268 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:40.852139 (+ 871us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:40.852206 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:40.852238 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:40.852374 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:40.852479 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 01:11:40.852592 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:11:40.853907 20722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:40.978045 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 20600
I20260619 01:11:40.999608 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 20249
I20260619 01:11:41.000756 20310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:41.111743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 20249
2026-06-19T01:11:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2897 ms)
[----------] 6 tests from EncryptionPolicyTest (22166 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:41 dist-test-slave-5p8w krb5kdc[20778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:41 dist-test-slave-5p8w krb5kdc[20778](info): set up 2 sockets
Jun 19 01:11:41 dist-test-slave-5p8w krb5kdc[20778](info): commencing operation
krb5kdc: starting...
W20260619 01:11:42.258277 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:42 dist-test-slave-5p8w krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:11:42.520589 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:42.861987 20791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:42.862341 20791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:42.862510 20791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:42.869783 20791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:42.869957 20791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:42.870066 20791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:42.870178 20791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:42.870260 20791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:42.880461 20791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:42.881886 20791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:42.883222 20791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:42.895689 20797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:42.895721 20796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:42.895761 20799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:42.896513 20791 server_base.cc:1034] running on GCE node
I20260619 01:11:42.897382 20791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:42.898837 20791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:42.900079 20791 hybrid_clock.cc:648] HybridClock initialized: now 1781831502900045 us; error 49 us; skew 500 ppm
I20260619 01:11:42.900461 20791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:42 dist-test-slave-5p8w krb5kdc[20778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:42.903699 20791 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:11:42.905614 20791 webserver.cc:460] Webserver started at http://127.0.0.1:42635/ using document root <none> and password file <none>
I20260619 01:11:42.906335 20791 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:42.906486 20791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:42.906798 20791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:42.909873 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "4e2aea578a9c42d6b81cffe9e0ad98cd"
format_stamp: "Formatted at 2026-06-19 01:11:42 on dist-test-slave-5p8w"
server_key: "206b028f5f629beb6d604df43f690a52"
server_key_iv: "ba114862eeaf4b275181f985cefa4319"
server_key_version: "encryptionkey@0"
I20260619 01:11:42.910668 20791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "4e2aea578a9c42d6b81cffe9e0ad98cd"
format_stamp: "Formatted at 2026-06-19 01:11:42 on dist-test-slave-5p8w"
server_key: "206b028f5f629beb6d604df43f690a52"
server_key_iv: "ba114862eeaf4b275181f985cefa4319"
server_key_version: "encryptionkey@0"
I20260619 01:11:42.916033 20791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 01:11:42.919610 20806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:42.920982 20791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:11:42.921185 20791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "4e2aea578a9c42d6b81cffe9e0ad98cd"
format_stamp: "Formatted at 2026-06-19 01:11:42 on dist-test-slave-5p8w"
server_key: "206b028f5f629beb6d604df43f690a52"
server_key_iv: "ba114862eeaf4b275181f985cefa4319"
server_key_version: "encryptionkey@0"
I20260619 01:11:42.921446 20791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:42.940209 20791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:42.944928 20791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:42.945243 20791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:42.963912 20791 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42041
I20260619 01:11:42.963932 20857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42041 every 8 connection(s)
I20260619 01:11:42.965377 20791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:42.966034 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 20791
I20260619 01:11:42.966269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:42.967016 388 external_mini_cluster.cc:1428] Setting key 0a4128a57548b1c1474a67de15432078
I20260619 01:11:42.970057 20858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:11:42 dist-test-slave-5p8w krb5kdc[20778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 01:11:42.981033 20858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd: Bootstrap starting.
I20260619 01:11:42.984731 20858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:42.985893 20858 log.cc:824] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:42.988930 20858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd: No bootstrap required, opened a new log
I20260619 01:11:42.990383 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:42.969051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56047 (local address 127.0.0.1:42041)
0619 01:11:42.969899 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:42.969912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:42.969953 (+ 41us) server_negotiation.cc:407] Connection header received
0619 01:11:42.971694 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:42.971719 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:42.972141 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:42.972560 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:42.974147 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:42.975522 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:42.976922 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:42.977688 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:42.977909 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:42.982182 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:42.982191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:42.982207 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:42.982248 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:42.985665 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:42.986712 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:42.986719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:42.986727 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:42.986897 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:42.987492 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:42.987496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:42.987498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:42.988023 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:42.988383 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:42.988891 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 01:11:42.989286 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":435,"threads_started":1}
I20260619 01:11:42.994853 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:11:42.997813 20858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2aea578a9c42d6b81cffe9e0ad98cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } }
I20260619 01:11:42.998303 20858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:42.998461 20858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2aea578a9c42d6b81cffe9e0ad98cd, State: Initialized, Role: FOLLOWER
I20260619 01:11:42.999048 20858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2aea578a9c42d6b81cffe9e0ad98cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } }
I20260619 01:11:42.999301 20858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:42.999487 20858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:42.999697 20858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:43.003682 20858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2aea578a9c42d6b81cffe9e0ad98cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } }
I20260619 01:11:43.004487 20858 leader_election.cc:302] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e2aea578a9c42d6b81cffe9e0ad98cd; no voters:
I20260619 01:11:43.005270 20858 leader_election.cc:288] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:43.005431 20868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:43.007014 20868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [term 1 LEADER]: Becoming Leader. State: Replica: 4e2aea578a9c42d6b81cffe9e0ad98cd, State: Running, Role: LEADER
I20260619 01:11:43.007866 20858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:43.007735 20868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2aea578a9c42d6b81cffe9e0ad98cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } }
I20260619 01:11:43.012416 20869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e2aea578a9c42d6b81cffe9e0ad98cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2aea578a9c42d6b81cffe9e0ad98cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } } }
I20260619 01:11:43.012869 20869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:43.016357 20870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2aea578a9c42d6b81cffe9e0ad98cd. Latest consensus state: current_term: 1 leader_uuid: "4e2aea578a9c42d6b81cffe9e0ad98cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2aea578a9c42d6b81cffe9e0ad98cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } } }
I20260619 01:11:43.016661 20876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:43.016712 20870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:43.021180 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:42.999733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39961 (local address 127.0.0.1:42041)
0619 01:11:43.000031 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:43.000041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:43.000598 (+ 557us) server_negotiation.cc:407] Connection header received
0619 01:11:43.000793 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:43.000798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:43.000862 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:43.000968 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:43.003079 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:43.003935 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:43.006208 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:43.006985 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:43.007125 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:43.011090 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:43.011102 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:43.011112 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:43.011182 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:43.016817 (+ 5635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:43.018179 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:43.018184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:43.018188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:43.018477 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:43.019206 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:43.019211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:43.019215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:43.019700 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:43.020072 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:43.020469 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 01:11:43.020710 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260619 01:11:43.022693 20876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:43.032943 20876 catalog_manager.cc:1347] Generated new cluster ID: 5b7c515ea19c43db9235fecbbdccd1ab
I20260619 01:11:43.033083 20876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:43.053889 20876 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:43.055303 20876 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:43.069629 20876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e2aea578a9c42d6b81cffe9e0ad98cd: Generated new TSK 0
I20260619 01:11:43.070570 20876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:43.085536 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 20791
I20260619 01:11:43.087495 20853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:43.187806 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 20791
W20260619 01:11:43.205556 20776 connection.cc:535] client connection to 127.0.0.1:42041 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42041): Connection reset by peer (error 104)
2026-06-19T01:11:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2064 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:43 dist-test-slave-5p8w krb5kdc[20890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:43 dist-test-slave-5p8w krb5kdc[20890](info): set up 2 sockets
Jun 19 01:11:43 dist-test-slave-5p8w krb5kdc[20890](info): commencing operation
krb5kdc: starting...
W20260619 01:11:44.301636 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:44 dist-test-slave-5p8w krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:11:44.560696 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:44.900669 20903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:44.900985 20903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:44.901161 20903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:44.909344 20903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:44.909507 20903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:44.909617 20903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:44.909725 20903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:44.909803 20903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:44.920266 20903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40029
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:44.921799 20903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:44.923192 20903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:44.935508 20909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:44.935626 20911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:44.935587 20908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:44.936443 20903 server_base.cc:1034] running on GCE node
I20260619 01:11:44.937250 20903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:44.938859 20903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:44.940101 20903 hybrid_clock.cc:648] HybridClock initialized: now 1781831504940043 us; error 54 us; skew 500 ppm
I20260619 01:11:44.940464 20903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:44 dist-test-slave-5p8w krb5kdc[20890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:44.943655 20903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:11:44.945173 20903 webserver.cc:460] Webserver started at http://127.0.0.1:46351/ using document root <none> and password file <none>
I20260619 01:11:44.945943 20903 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:44.946103 20903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:44.946425 20903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:44.949597 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "c78d5a4609b343958ad5e4f99b194f22"
format_stamp: "Formatted at 2026-06-19 01:11:44 on dist-test-slave-5p8w"
server_key: "9053d68a4201361efd524adfa2d37ab7"
server_key_iv: "6bd57b6a70a3e0e516ff33e0eabe1420"
server_key_version: "encryptionkey@0"
I20260619 01:11:44.950471 20903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "c78d5a4609b343958ad5e4f99b194f22"
format_stamp: "Formatted at 2026-06-19 01:11:44 on dist-test-slave-5p8w"
server_key: "9053d68a4201361efd524adfa2d37ab7"
server_key_iv: "6bd57b6a70a3e0e516ff33e0eabe1420"
server_key_version: "encryptionkey@0"
I20260619 01:11:44.956243 20903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:11:44.960119 20918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:44.961720 20903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:44.961928 20903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "c78d5a4609b343958ad5e4f99b194f22"
format_stamp: "Formatted at 2026-06-19 01:11:44 on dist-test-slave-5p8w"
server_key: "9053d68a4201361efd524adfa2d37ab7"
server_key_iv: "6bd57b6a70a3e0e516ff33e0eabe1420"
server_key_version: "encryptionkey@0"
I20260619 01:11:44.962163 20903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:44.998656 20903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:45.003104 20903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:45.003422 20903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:45.022828 20903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40029
I20260619 01:11:45.022853 20969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40029 every 8 connection(s)
I20260619 01:11:45.024376 20903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:45.027109 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 20903
I20260619 01:11:45.027343 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:45.028003 388 external_mini_cluster.cc:1428] Setting key ba79fca0682b1c34d77860f588f9509d
I20260619 01:11:45.029628 20970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:45.042128 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:45.030123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52727 (local address 127.0.0.1:40029)
0619 01:11:45.030962 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:45.030974 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:45.031018 (+ 44us) server_negotiation.cc:407] Connection header received
0619 01:11:45.032694 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:45.032720 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:45.033171 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:45.033718 (+ 547us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:45.034300 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:45.035721 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:45.037187 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:45.037997 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:45.038210 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:45.039601 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:45.039610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:45.039626 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:45.040241 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:45.040327 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:45.040357 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:45.040557 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:45.040819 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 01:11:45.041007 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":383,"threads_started":1}
I20260619 01:11:45.042560 20970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22: Bootstrap starting.
I20260619 01:11:45.046286 20970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:45.046453 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:11:45.047441 20970 log.cc:824] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:45.050921 20970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22: No bootstrap required, opened a new log
I20260619 01:11:45.058820 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:45.051590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52497 (local address 127.0.0.1:40029)
0619 01:11:45.051802 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:45.051811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:45.052204 (+ 393us) server_negotiation.cc:407] Connection header received
0619 01:11:45.052383 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:45.052389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:45.052469 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:45.052600 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:45.053136 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:45.054386 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:45.055878 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:45.056666 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:45.056817 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:45.057997 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:45.058004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:45.058009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:45.058038 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:45.058102 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:45.058124 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:45.058253 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:45.058387 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 01:11:45.058447 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:11:45.061269 20970 raft_consensus.cc:348] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78d5a4609b343958ad5e4f99b194f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40029 } }
I20260619 01:11:45.061749 20970 raft_consensus.cc:374] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:45.061872 20970 raft_consensus.cc:729] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78d5a4609b343958ad5e4f99b194f22, State: Initialized, Role: FOLLOWER
I20260619 01:11:45.062490 20970 consensus_queue.cc:260] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78d5a4609b343958ad5e4f99b194f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40029 } }
I20260619 01:11:45.062741 20970 raft_consensus.cc:388] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:45.062911 20970 raft_consensus.cc:482] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:45.063102 20970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:45.065809 20970 raft_consensus.cc:504] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78d5a4609b343958ad5e4f99b194f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40029 } }
I20260619 01:11:45.066443 20970 leader_election.cc:302] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c78d5a4609b343958ad5e4f99b194f22; no voters:
I20260619 01:11:45.067035 20970 leader_election.cc:288] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:45.067248 20980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:45.068040 20980 raft_consensus.cc:686] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [term 1 LEADER]: Becoming Leader. State: Replica: c78d5a4609b343958ad5e4f99b194f22, State: Running, Role: LEADER
I20260619 01:11:45.068584 20980 consensus_queue.cc:237] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78d5a4609b343958ad5e4f99b194f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40029 } }
I20260619 01:11:45.069494 20970 sys_catalog.cc:564] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:45.071604 20982 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c78d5a4609b343958ad5e4f99b194f22. Latest consensus state: current_term: 1 leader_uuid: "c78d5a4609b343958ad5e4f99b194f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78d5a4609b343958ad5e4f99b194f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40029 } } }
I20260619 01:11:45.071708 20981 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c78d5a4609b343958ad5e4f99b194f22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78d5a4609b343958ad5e4f99b194f22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40029 } } }
I20260619 01:11:45.072114 20982 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:45.072196 20981 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:45.073268 20987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:45.079736 20987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:45.088584 20987 catalog_manager.cc:1347] Generated new cluster ID: d41ccbc12bfb4c19a739a264afefa2ee
I20260619 01:11:45.088721 20987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:45.105211 20987 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:45.107203 20987 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:45.118408 20987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c78d5a4609b343958ad5e4f99b194f22: Generated new TSK 0
I20260619 01:11:45.119634 20987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:45.140291 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 20903
W20260619 01:11:45.141721 20965 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52727
W20260619 01:11:45.142549 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 20903: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:11:45.142726 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 20903
W20260619 01:11:45.156605 20888 connection.cc:535] client connection to 127.0.0.1:40029 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40029): Connection reset by peer (error 104)
2026-06-19T01:11:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:45 dist-test-slave-5p8w krb5kdc[21001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:45 dist-test-slave-5p8w krb5kdc[21001](info): set up 2 sockets
Jun 19 01:11:45 dist-test-slave-5p8w krb5kdc[21001](info): commencing operation
krb5kdc: starting...
W20260619 01:11:46.241775 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:46 dist-test-slave-5p8w krb5kdc[21001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:11:46.506070 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:46.846817 21014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:46.847126 21014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:46.847308 21014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:46.855265 21014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:46.855394 21014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:46.855501 21014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:46.855604 21014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:46.855686 21014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:46.866031 21014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:46.867432 21014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:46.868757 21014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:46.880831 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:46.880753 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:46.880803 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:46.881443 21014 server_base.cc:1034] running on GCE node
I20260619 01:11:46.882313 21014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:46.883760 21014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:46.885010 21014 hybrid_clock.cc:648] HybridClock initialized: now 1781831506884962 us; error 48 us; skew 500 ppm
I20260619 01:11:46.885433 21014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:46 dist-test-slave-5p8w krb5kdc[21001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:46.893760 21014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:11:46.895330 21014 webserver.cc:460] Webserver started at http://127.0.0.1:41503/ using document root <none> and password file <none>
I20260619 01:11:46.896082 21014 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:46.896241 21014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:46.896577 21014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:46.899770 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "c29c3d542f784a929eadaadab5e76fdd"
format_stamp: "Formatted at 2026-06-19 01:11:46 on dist-test-slave-5p8w"
server_key: "f6fd30e57dedcccbf947db7dbf40233f"
server_key_iv: "e181bb1af774499e9e298eaf52828c03"
server_key_version: "encryptionkey@0"
I20260619 01:11:46.900671 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "c29c3d542f784a929eadaadab5e76fdd"
format_stamp: "Formatted at 2026-06-19 01:11:46 on dist-test-slave-5p8w"
server_key: "f6fd30e57dedcccbf947db7dbf40233f"
server_key_iv: "e181bb1af774499e9e298eaf52828c03"
server_key_version: "encryptionkey@0"
I20260619 01:11:46.906451 21014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 01:11:46.910105 21029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:46.911592 21014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:11:46.911777 21014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "c29c3d542f784a929eadaadab5e76fdd"
format_stamp: "Formatted at 2026-06-19 01:11:46 on dist-test-slave-5p8w"
server_key: "f6fd30e57dedcccbf947db7dbf40233f"
server_key_iv: "e181bb1af774499e9e298eaf52828c03"
server_key_version: "encryptionkey@0"
I20260619 01:11:46.911988 21014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:46.917909 21014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:46.922030 21014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:46.922340 21014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:46.940310 21014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37835
I20260619 01:11:46.940335 21080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37835 every 8 connection(s)
I20260619 01:11:46.941968 21014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:46.942324 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 21014
I20260619 01:11:46.942528 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:46.943215 388 external_mini_cluster.cc:1428] Setting key dcd71acf57c7e6e1d36df157956a0915
I20260619 01:11:46.946610 21081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:46.951437 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:46.945032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53855 (local address 127.0.0.1:37835)
0619 01:11:46.945887 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:46.945900 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:46.945941 (+ 41us) server_negotiation.cc:407] Connection header received
0619 01:11:46.947788 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:46.947815 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:46.948182 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:46.948531 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:46.949732 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:46.949739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:46.949754 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:46.949769 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:46.949851 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:46.949875 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:46.950055 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:46.950174 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 01:11:46.950401 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":401,"threads_started":1}
I20260619 01:11:46.955449 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:11:46.959801 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd: Bootstrap starting.
I20260619 01:11:46.963312 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:46.960750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54121 (local address 127.0.0.1:37835)
0619 01:11:46.960966 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:46.960973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:46.961278 (+ 305us) server_negotiation.cc:407] Connection header received
0619 01:11:46.961500 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:46.961503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:46.961565 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:46.961667 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:46.962663 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:46.962667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:46.962670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:46.962684 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:46.962721 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:46.962735 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:46.962826 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:46.962965 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 01:11:46.963010 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 01:11:46.963934 21081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:46.965065 21081 log.cc:824] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:46.967788 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd: No bootstrap required, opened a new log
I20260619 01:11:46.975155 21081 raft_consensus.cc:348] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c3d542f784a929eadaadab5e76fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37835 } }
I20260619 01:11:46.975636 21081 raft_consensus.cc:374] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:46.975760 21081 raft_consensus.cc:729] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29c3d542f784a929eadaadab5e76fdd, State: Initialized, Role: FOLLOWER
I20260619 01:11:46.976367 21081 consensus_queue.cc:260] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c3d542f784a929eadaadab5e76fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37835 } }
I20260619 01:11:46.976611 21081 raft_consensus.cc:388] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:46.976768 21081 raft_consensus.cc:482] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:46.976940 21081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:46.979223 21081 raft_consensus.cc:504] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c3d542f784a929eadaadab5e76fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37835 } }
I20260619 01:11:46.979777 21081 leader_election.cc:302] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c29c3d542f784a929eadaadab5e76fdd; no voters:
I20260619 01:11:46.980402 21081 leader_election.cc:288] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:46.980567 21091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:46.981819 21091 raft_consensus.cc:686] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [term 1 LEADER]: Becoming Leader. State: Replica: c29c3d542f784a929eadaadab5e76fdd, State: Running, Role: LEADER
I20260619 01:11:46.982343 21091 consensus_queue.cc:237] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c3d542f784a929eadaadab5e76fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37835 } }
I20260619 01:11:46.982621 21081 sys_catalog.cc:564] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:46.985396 21092 sys_catalog.cc:455] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c29c3d542f784a929eadaadab5e76fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c3d542f784a929eadaadab5e76fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37835 } } }
I20260619 01:11:46.985924 21092 sys_catalog.cc:458] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:46.985368 21093 sys_catalog.cc:455] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c29c3d542f784a929eadaadab5e76fdd. Latest consensus state: current_term: 1 leader_uuid: "c29c3d542f784a929eadaadab5e76fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c29c3d542f784a929eadaadab5e76fdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37835 } } }
I20260619 01:11:46.987737 21093 sys_catalog.cc:458] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:46.989357 21100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:46.994647 21100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:47.003859 21100 catalog_manager.cc:1347] Generated new cluster ID: e73506dd2b7241c28652d8fc9677b497
I20260619 01:11:47.003997 21100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:47.023494 21100 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:47.023953 21100 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:47.031975 21100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c29c3d542f784a929eadaadab5e76fdd: Generated new TSK 0
I20260619 01:11:47.032799 21100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:47.044209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21014
W20260619 01:11:47.045251 21076 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53855
W20260619 01:11:47.046095 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21014: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:11:47.046274 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 21014
2026-06-19T01:11:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:47 dist-test-slave-5p8w krb5kdc[21112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:47 dist-test-slave-5p8w krb5kdc[21112](info): set up 2 sockets
Jun 19 01:11:47 dist-test-slave-5p8w krb5kdc[21112](info): commencing operation
krb5kdc: starting...
W20260619 01:11:48.141883 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:48 dist-test-slave-5p8w krb5kdc[21112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:11:48.399183 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43001
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:48.735276 21125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:48.735594 21125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:48.735726 21125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:48.743006 21125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:48.743124 21125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:48.743237 21125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:48.743345 21125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:48.743435 21125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:48.753459 21125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:48.754853 21125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:48.756224 21125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:48.768534 21130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:48.768479 21131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:48.768594 21133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:48.769637 21125 server_base.cc:1034] running on GCE node
I20260619 01:11:48.770452 21125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:48.771911 21125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:48.773176 21125 hybrid_clock.cc:648] HybridClock initialized: now 1781831508773127 us; error 60 us; skew 500 ppm
I20260619 01:11:48.773602 21125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:48 dist-test-slave-5p8w krb5kdc[21112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:48.776861 21125 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:11:48.778574 21125 webserver.cc:460] Webserver started at http://127.0.0.1:42095/ using document root <none> and password file <none>
I20260619 01:11:48.779320 21125 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:48.779472 21125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:48.779799 21125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:48.782961 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "4f32ece82e6b463a88ae05018c336b76"
format_stamp: "Formatted at 2026-06-19 01:11:48 on dist-test-slave-5p8w"
server_key: "841dc17fe9d3826028bf977732001e00"
server_key_iv: "4ed370951cab7d47199353a592bbe198"
server_key_version: "encryptionkey@0"
I20260619 01:11:48.783762 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "4f32ece82e6b463a88ae05018c336b76"
format_stamp: "Formatted at 2026-06-19 01:11:48 on dist-test-slave-5p8w"
server_key: "841dc17fe9d3826028bf977732001e00"
server_key_iv: "4ed370951cab7d47199353a592bbe198"
server_key_version: "encryptionkey@0"
I20260619 01:11:48.789100 21125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 01:11:48.792927 21140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:48.794548 21125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:48.794770 21125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "4f32ece82e6b463a88ae05018c336b76"
format_stamp: "Formatted at 2026-06-19 01:11:48 on dist-test-slave-5p8w"
server_key: "841dc17fe9d3826028bf977732001e00"
server_key_iv: "4ed370951cab7d47199353a592bbe198"
server_key_version: "encryptionkey@0"
I20260619 01:11:48.795013 21125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:48.810040 21125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:48.814227 21125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:48.814551 21125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:48.834229 21125 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43001
I20260619 01:11:48.834249 21191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43001 every 8 connection(s)
I20260619 01:11:48.835804 21125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:48.839767 21192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:48.844264 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 21125
I20260619 01:11:48.844497 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:48.844974 388 external_mini_cluster.cc:1428] Setting key ae37eb55c3f9a84a0295bd5d182a342a
I20260619 01:11:48.851388 21192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76: Bootstrap starting.
I20260619 01:11:48.855408 21192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:48.856627 21192 log.cc:824] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:48.859954 21192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76: No bootstrap required, opened a new log
Jun 19 01:11:48 dist-test-slave-5p8w krb5kdc[21112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 01:11:48.869544 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:48.846823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50399 (local address 127.0.0.1:43001)
0619 01:11:48.847683 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:48.847699 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:48.847741 (+ 42us) server_negotiation.cc:407] Connection header received
0619 01:11:48.849670 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:48.849695 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:48.850131 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:48.850494 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:48.852475 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:48.854059 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:48.855980 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:48.856716 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:48.856943 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:48.861569 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:48.861577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:48.861594 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:48.861639 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:48.864861 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:48.865854 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:48.865861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:48.865871 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:48.866076 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:48.866510 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:48.866513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:48.866515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:48.867146 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:48.867369 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:48.867915 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 01:11:48.868356 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":404,"threads_started":1}
I20260619 01:11:48.869982 21192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f32ece82e6b463a88ae05018c336b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260619 01:11:48.870469 21192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:48.870579 21192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f32ece82e6b463a88ae05018c336b76, State: Initialized, Role: FOLLOWER
I20260619 01:11:48.871246 21192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f32ece82e6b463a88ae05018c336b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260619 01:11:48.871522 21192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:48.871678 21192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:48.871870 21192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:48.873975 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:11:48.874775 21192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f32ece82e6b463a88ae05018c336b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260619 01:11:48.875305 21192 leader_election.cc:302] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f32ece82e6b463a88ae05018c336b76; no voters:
I20260619 01:11:48.875923 21192 leader_election.cc:288] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:48.876116 21197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:48.877859 21197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [term 1 LEADER]: Becoming Leader. State: Replica: 4f32ece82e6b463a88ae05018c336b76, State: Running, Role: LEADER
I20260619 01:11:48.878329 21192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:48.878749 21197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f32ece82e6b463a88ae05018c336b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } }
I20260619 01:11:48.882403 21198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f32ece82e6b463a88ae05018c336b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f32ece82e6b463a88ae05018c336b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } } }
I20260619 01:11:48.883067 21198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:48.882473 21199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f32ece82e6b463a88ae05018c336b76. Latest consensus state: current_term: 1 leader_uuid: "4f32ece82e6b463a88ae05018c336b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f32ece82e6b463a88ae05018c336b76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43001 } } }
I20260619 01:11:48.883889 21199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:48.890581 21207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:48.896888 21207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:48.908833 21207 catalog_manager.cc:1347] Generated new cluster ID: 5a365d33ea20457d99f35e3ce0050359
I20260619 01:11:48.909042 21207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:48.910435 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:48.892704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37567 (local address 127.0.0.1:43001)
0619 01:11:48.892903 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:48.892911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:48.893191 (+ 280us) server_negotiation.cc:407] Connection header received
0619 01:11:48.893483 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:48.893492 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:48.893587 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:48.893720 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:48.895689 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:48.896840 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:48.898836 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:48.899595 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:48.899695 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:48.902853 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:48.902860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:48.902866 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:48.902913 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:48.906882 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:48.907885 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:48.907891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:48.907898 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:48.908056 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:48.908576 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:48.908581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:48.908585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:48.909051 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:48.909381 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:48.909773 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 01:11:48.910041 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 01:11:48.926618 21207 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:48.928220 21207 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:48.937041 21207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f32ece82e6b463a88ae05018c336b76: Generated new TSK 0
I20260619 01:11:48.937964 21207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:48.947090 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21125
I20260619 01:11:48.948302 21187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:49.040125 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 21125
2026-06-19T01:11:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:49 dist-test-slave-5p8w krb5kdc[21224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:49 dist-test-slave-5p8w krb5kdc[21224](info): set up 2 sockets
Jun 19 01:11:49 dist-test-slave-5p8w krb5kdc[21224](info): commencing operation
krb5kdc: starting...
W20260619 01:11:50.153837 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:50 dist-test-slave-5p8w krb5kdc[21224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:11:50.416752 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:50.759984 21237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:50.760316 21237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:50.760488 21237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:50.768126 21237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:50.768266 21237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:50.768359 21237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:50.768471 21237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:50.768550 21237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:50.779060 21237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:50.780479 21237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:50.781944 21237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:50.794198 21242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:50.794175 21245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:50.794164 21243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:50.795080 21237 server_base.cc:1034] running on GCE node
I20260619 01:11:50.795928 21237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:50.797643 21237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:50.798895 21237 hybrid_clock.cc:648] HybridClock initialized: now 1781831510798835 us; error 34 us; skew 500 ppm
I20260619 01:11:50.799345 21237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:50 dist-test-slave-5p8w krb5kdc[21224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:50.802953 21237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:11:50.804692 21237 webserver.cc:460] Webserver started at http://127.0.0.1:39653/ using document root <none> and password file <none>
I20260619 01:11:50.805565 21237 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:50.805732 21237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:50.806097 21237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:50.809592 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "ac10ddfb63bd4c31b48733c686d38c63"
format_stamp: "Formatted at 2026-06-19 01:11:50 on dist-test-slave-5p8w"
server_key: "7a086e3a4597ca8a1b94255c9818bb66"
server_key_iv: "594eccad7e726cdfc95da026aae1d981"
server_key_version: "encryptionkey@0"
I20260619 01:11:50.810514 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "ac10ddfb63bd4c31b48733c686d38c63"
format_stamp: "Formatted at 2026-06-19 01:11:50 on dist-test-slave-5p8w"
server_key: "7a086e3a4597ca8a1b94255c9818bb66"
server_key_iv: "594eccad7e726cdfc95da026aae1d981"
server_key_version: "encryptionkey@0"
I20260619 01:11:50.816622 21237 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260619 01:11:50.820801 21252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:50.822392 21237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:50.822613 21237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "ac10ddfb63bd4c31b48733c686d38c63"
format_stamp: "Formatted at 2026-06-19 01:11:50 on dist-test-slave-5p8w"
server_key: "7a086e3a4597ca8a1b94255c9818bb66"
server_key_iv: "594eccad7e726cdfc95da026aae1d981"
server_key_version: "encryptionkey@0"
I20260619 01:11:50.822839 21237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:50.844758 21237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:50.849737 21237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:50.850080 21237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:50.869837 21237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46629
I20260619 01:11:50.869867 21303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46629 every 8 connection(s)
I20260619 01:11:50.871308 21237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:50.873449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 21237
I20260619 01:11:50.873631 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:50.874238 388 external_mini_cluster.cc:1428] Setting key 502244106fbde0a031be0f76b232914c
I20260619 01:11:50.876065 21304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:50.888448 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:50.876298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37321 (local address 127.0.0.1:46629)
0619 01:11:50.877164 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:50.877177 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:50.877218 (+ 41us) server_negotiation.cc:407] Connection header received
0619 01:11:50.878994 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:50.879018 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:50.879461 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:50.879820 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:50.880508 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:50.881977 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:50.883882 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:50.884561 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:50.884697 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:50.886064 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:50.886072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:50.886086 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:50.886664 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:50.886744 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:50.886791 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:50.886880 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:50.887127 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 01:11:50.887420 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":374,"threads_started":1}
I20260619 01:11:50.889729 21304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63: Bootstrap starting.
I20260619 01:11:50.892392 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:11:50.893195 21304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:50.894228 21304 log.cc:824] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:50.897782 21304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63: No bootstrap required, opened a new log
I20260619 01:11:50.903709 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:50.896818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44223 (local address 127.0.0.1:46629)
0619 01:11:50.897004 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:50.897011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:50.897520 (+ 509us) server_negotiation.cc:407] Connection header received
0619 01:11:50.897734 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:50.897739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:50.897801 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:50.897919 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:50.898522 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:50.899245 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:50.901182 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:50.901715 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:50.901814 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:50.902996 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:50.903001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:50.903004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:50.903023 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:50.903058 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:50.903079 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:50.903155 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:50.903300 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 01:11:50.903374 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 01:11:50.907176 21304 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac10ddfb63bd4c31b48733c686d38c63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } }
I20260619 01:11:50.907572 21304 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:50.907678 21304 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac10ddfb63bd4c31b48733c686d38c63, State: Initialized, Role: FOLLOWER
I20260619 01:11:50.908289 21304 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac10ddfb63bd4c31b48733c686d38c63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } }
I20260619 01:11:50.908525 21304 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:50.908679 21304 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:50.908875 21304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:50.911305 21304 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac10ddfb63bd4c31b48733c686d38c63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } }
I20260619 01:11:50.911820 21304 leader_election.cc:302] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac10ddfb63bd4c31b48733c686d38c63; no voters:
I20260619 01:11:50.912434 21304 leader_election.cc:288] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:50.912683 21314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:50.913427 21314 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [term 1 LEADER]: Becoming Leader. State: Replica: ac10ddfb63bd4c31b48733c686d38c63, State: Running, Role: LEADER
I20260619 01:11:50.914029 21314 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac10ddfb63bd4c31b48733c686d38c63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } }
I20260619 01:11:50.914616 21304 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:50.916935 21316 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac10ddfb63bd4c31b48733c686d38c63. Latest consensus state: current_term: 1 leader_uuid: "ac10ddfb63bd4c31b48733c686d38c63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac10ddfb63bd4c31b48733c686d38c63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } } }
I20260619 01:11:50.917395 21316 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:50.917385 21315 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac10ddfb63bd4c31b48733c686d38c63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac10ddfb63bd4c31b48733c686d38c63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } } }
I20260619 01:11:50.917711 21315 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:50.919482 21319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:50.925344 21319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:50.935424 21319 catalog_manager.cc:1347] Generated new cluster ID: adeeb01961b4447899a7d639fe8573e2
I20260619 01:11:50.935561 21319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:50.962600 21319 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:50.964570 21319 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:50.975796 21319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac10ddfb63bd4c31b48733c686d38c63: Generated new TSK 0
I20260619 01:11:50.976985 21319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:50.993229 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21237
W20260619 01:11:50.994527 21299 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37321
W20260619 01:11:50.995378 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21237: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:11:50.995548 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 21237
2026-06-19T01:11:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:51 dist-test-slave-5p8w krb5kdc[21335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:51 dist-test-slave-5p8w krb5kdc[21335](info): set up 2 sockets
Jun 19 01:11:51 dist-test-slave-5p8w krb5kdc[21335](info): commencing operation
krb5kdc: starting...
W20260619 01:11:52.113897 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:52 dist-test-slave-5p8w krb5kdc[21335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:11:52.376669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:52.722859 21348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:52.723330 21348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:52.723489 21348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:52.731472 21348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:52.731655 21348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:52.731736 21348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:52.731823 21348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:52.731899 21348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:52.742281 21348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:52.743644 21348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:52.744958 21348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:52.756945 21353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:52.756968 21354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:52.757014 21356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:52.758378 21348 server_base.cc:1034] running on GCE node
I20260619 01:11:52.759275 21348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:52.760769 21348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:52.761978 21348 hybrid_clock.cc:648] HybridClock initialized: now 1781831512761934 us; error 37 us; skew 500 ppm
I20260619 01:11:52.762333 21348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:52 dist-test-slave-5p8w krb5kdc[21335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:52.770105 21348 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:11:52.771714 21348 webserver.cc:460] Webserver started at http://127.0.0.1:40175/ using document root <none> and password file <none>
I20260619 01:11:52.772413 21348 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:52.772513 21348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:52.772783 21348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:52.775805 21348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "713efbb67a464fcb9cfa81c7aed3d4e3"
format_stamp: "Formatted at 2026-06-19 01:11:52 on dist-test-slave-5p8w"
server_key: "d51fa801f17acd090a69cd315611dbb3"
server_key_iv: "75a35402102430b96afa100b437c7c94"
server_key_version: "encryptionkey@0"
I20260619 01:11:52.776521 21348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "713efbb67a464fcb9cfa81c7aed3d4e3"
format_stamp: "Formatted at 2026-06-19 01:11:52 on dist-test-slave-5p8w"
server_key: "d51fa801f17acd090a69cd315611dbb3"
server_key_iv: "75a35402102430b96afa100b437c7c94"
server_key_version: "encryptionkey@0"
I20260619 01:11:52.781818 21348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260619 01:11:52.785125 21363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:52.786551 21348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:11:52.786746 21348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "713efbb67a464fcb9cfa81c7aed3d4e3"
format_stamp: "Formatted at 2026-06-19 01:11:52 on dist-test-slave-5p8w"
server_key: "d51fa801f17acd090a69cd315611dbb3"
server_key_iv: "75a35402102430b96afa100b437c7c94"
server_key_version: "encryptionkey@0"
I20260619 01:11:52.786947 21348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:52.792837 21348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:52.796758 21348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:52.797052 21348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:52.815845 21348 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46569
I20260619 01:11:52.815886 21414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46569 every 8 connection(s)
I20260619 01:11:52.817345 21348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:52.821017 21415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:52.822523 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 21348
I20260619 01:11:52.822706 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:52.823130 388 external_mini_cluster.cc:1428] Setting key ff35822bdb50e7232043e71b7c3bf199
I20260619 01:11:52.830986 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:52.824994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36365 (local address 127.0.0.1:46569)
0619 01:11:52.825840 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:52.825851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:52.825896 (+ 45us) server_negotiation.cc:407] Connection header received
0619 01:11:52.827493 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:52.827520 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:52.827905 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:52.828261 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:52.829423 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:52.829430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:52.829445 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:52.829460 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:52.829519 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:52.829543 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:52.829732 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:52.829834 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 01:11:52.830044 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":381,"threads_started":1}
I20260619 01:11:52.834721 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:11:52.834766 21415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3: Bootstrap starting.
I20260619 01:11:52.837877 21415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:52.838985 21415 log.cc:824] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:52.841797 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:52.839085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39293 (local address 127.0.0.1:46569)
0619 01:11:52.839317 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:52.839326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:52.839814 (+ 488us) server_negotiation.cc:407] Connection header received
0619 01:11:52.839964 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:52.839972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:52.840041 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:52.840170 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:52.841099 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:52.841104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:52.841109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:52.841127 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:52.841173 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:52.841189 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:52.841323 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:52.841455 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 01:11:52.841518 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:11:52.842170 21415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3: No bootstrap required, opened a new log
I20260619 01:11:52.849931 21415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713efbb67a464fcb9cfa81c7aed3d4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46569 } }
I20260619 01:11:52.850353 21415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:52.850461 21415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713efbb67a464fcb9cfa81c7aed3d4e3, State: Initialized, Role: FOLLOWER
I20260619 01:11:52.851158 21415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713efbb67a464fcb9cfa81c7aed3d4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46569 } }
I20260619 01:11:52.851449 21415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:52.851611 21415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:52.851760 21415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:52.853997 21415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713efbb67a464fcb9cfa81c7aed3d4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46569 } }
I20260619 01:11:52.854460 21415 leader_election.cc:302] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 713efbb67a464fcb9cfa81c7aed3d4e3; no voters:
I20260619 01:11:52.854976 21415 leader_election.cc:288] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:52.855150 21425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:52.855832 21425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [term 1 LEADER]: Becoming Leader. State: Replica: 713efbb67a464fcb9cfa81c7aed3d4e3, State: Running, Role: LEADER
I20260619 01:11:52.856331 21425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713efbb67a464fcb9cfa81c7aed3d4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46569 } }
I20260619 01:11:52.857128 21415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:52.859074 21427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 713efbb67a464fcb9cfa81c7aed3d4e3. Latest consensus state: current_term: 1 leader_uuid: "713efbb67a464fcb9cfa81c7aed3d4e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713efbb67a464fcb9cfa81c7aed3d4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46569 } } }
I20260619 01:11:52.859493 21427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:52.860190 21426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "713efbb67a464fcb9cfa81c7aed3d4e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "713efbb67a464fcb9cfa81c7aed3d4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46569 } } }
I20260619 01:11:52.860528 21430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:52.860622 21426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:52.867038 21430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:52.876513 21430 catalog_manager.cc:1347] Generated new cluster ID: fea6c35245444420a60e6be0d3f64967
I20260619 01:11:52.876734 21430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:52.887367 21430 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:52.887898 21430 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:52.901082 21430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 713efbb67a464fcb9cfa81c7aed3d4e3: Generated new TSK 0
I20260619 01:11:52.902009 21430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:52.913112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21348
W20260619 01:11:52.914404 21410 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36365
W20260619 01:11:52.915210 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21348: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:11:52.915377 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 21348
2026-06-19T01:11:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:53 dist-test-slave-5p8w krb5kdc[21446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:53 dist-test-slave-5p8w krb5kdc[21446](info): set up 2 sockets
Jun 19 01:11:53 dist-test-slave-5p8w krb5kdc[21446](info): commencing operation
krb5kdc: starting...
W20260619 01:11:54.021826 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:54 dist-test-slave-5p8w krb5kdc[21446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:11:54.280575 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:54.626643 21459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:54.626972 21459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:54.627075 21459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:54.634512 21459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:54.634640 21459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:54.634692 21459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:54.634752 21459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:54.634795 21459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:54.644901 21459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:54.646350 21459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:54.647611 21459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:54.659894 21467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:54.659974 21464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:54.659978 21465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:54.661191 21459 server_base.cc:1034] running on GCE node
I20260619 01:11:54.662081 21459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:54.663689 21459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:54.664897 21459 hybrid_clock.cc:648] HybridClock initialized: now 1781831514664851 us; error 38 us; skew 500 ppm
I20260619 01:11:54.665280 21459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:54 dist-test-slave-5p8w krb5kdc[21446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:54.668551 21459 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:11:54.670338 21459 webserver.cc:460] Webserver started at http://127.0.0.1:36793/ using document root <none> and password file <none>
I20260619 01:11:54.671151 21459 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:54.671307 21459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:54.671643 21459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:54.674943 21459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "4538a08ca14e4c9bbd95f19e7b510097"
format_stamp: "Formatted at 2026-06-19 01:11:54 on dist-test-slave-5p8w"
server_key: "5986ebdcfe6716456e2eb7aafe5482b2"
server_key_iv: "ee35b56657c9348bb89850a15a9267ff"
server_key_version: "encryptionkey@0"
I20260619 01:11:54.675753 21459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "4538a08ca14e4c9bbd95f19e7b510097"
format_stamp: "Formatted at 2026-06-19 01:11:54 on dist-test-slave-5p8w"
server_key: "5986ebdcfe6716456e2eb7aafe5482b2"
server_key_iv: "ee35b56657c9348bb89850a15a9267ff"
server_key_version: "encryptionkey@0"
I20260619 01:11:54.681571 21459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 01:11:54.685508 21474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:54.686966 21459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:11:54.687182 21459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "4538a08ca14e4c9bbd95f19e7b510097"
format_stamp: "Formatted at 2026-06-19 01:11:54 on dist-test-slave-5p8w"
server_key: "5986ebdcfe6716456e2eb7aafe5482b2"
server_key_iv: "ee35b56657c9348bb89850a15a9267ff"
server_key_version: "encryptionkey@0"
I20260619 01:11:54.687398 21459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:54.706679 21459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:54.710996 21459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:54.711309 21459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:54.730422 21459 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34371
I20260619 01:11:54.730448 21525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34371 every 8 connection(s)
I20260619 01:11:54.731899 21459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:54.735944 21526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:54.736506 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 21459
I20260619 01:11:54.736697 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:54.737111 388 external_mini_cluster.cc:1428] Setting key 73acc1f6d44d3c6f44049d80d47ea898
I20260619 01:11:54.751125 21526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097: Bootstrap starting.
Jun 19 01:11:54 dist-test-slave-5p8w krb5kdc[21446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 01:11:54.754763 21526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:54.756204 21526 log.cc:824] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:54.759773 21526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097: No bootstrap required, opened a new log
I20260619 01:11:54.763613 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:54.738976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:59935 (local address 127.0.0.1:34371)
0619 01:11:54.739804 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:54.739821 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:54.739871 (+ 50us) server_negotiation.cc:407] Connection header received
0619 01:11:54.742808 (+ 2937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:54.742846 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:54.743516 (+ 670us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:54.744040 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:54.745535 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:54.747615 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:54.749003 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:54.750158 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:54.750349 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:54.754710 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:54.754724 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:54.754753 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:54.754813 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:54.758721 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:54.759595 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:54.759604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:54.759612 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:54.759833 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:54.760295 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:54.760301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:54.760305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:54.761024 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:54.761378 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:54.761774 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 01:11:54.762167 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":336,"threads_started":1}
I20260619 01:11:54.767748 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:11:54.769658 21526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4538a08ca14e4c9bbd95f19e7b510097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260619 01:11:54.770079 21526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:54.770205 21526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4538a08ca14e4c9bbd95f19e7b510097, State: Initialized, Role: FOLLOWER
I20260619 01:11:54.770838 21526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4538a08ca14e4c9bbd95f19e7b510097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260619 01:11:54.771090 21526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:54.771246 21526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:54.771420 21526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:54.774279 21526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4538a08ca14e4c9bbd95f19e7b510097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260619 01:11:54.774837 21526 leader_election.cc:302] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4538a08ca14e4c9bbd95f19e7b510097; no voters:
I20260619 01:11:54.775513 21526 leader_election.cc:288] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:54.775625 21536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:54.777453 21536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [term 1 LEADER]: Becoming Leader. State: Replica: 4538a08ca14e4c9bbd95f19e7b510097, State: Running, Role: LEADER
I20260619 01:11:54.778189 21526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:54.778158 21536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4538a08ca14e4c9bbd95f19e7b510097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } }
I20260619 01:11:54.782466 21538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4538a08ca14e4c9bbd95f19e7b510097. Latest consensus state: current_term: 1 leader_uuid: "4538a08ca14e4c9bbd95f19e7b510097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4538a08ca14e4c9bbd95f19e7b510097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } } }
I20260619 01:11:54.784102 21538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:54.783165 21537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4538a08ca14e4c9bbd95f19e7b510097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4538a08ca14e4c9bbd95f19e7b510097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34371 } } }
I20260619 01:11:54.785499 21537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097 [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:54.787849 21547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:54.789778 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:54.772312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:36239 (local address 127.0.0.1:34371)
0619 01:11:54.772505 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:54.772514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:54.772890 (+ 376us) server_negotiation.cc:407] Connection header received
0619 01:11:54.773114 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:54.773120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:54.773195 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:54.773341 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:54.774698 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:54.775558 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:54.777211 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:54.777914 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:54.778052 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:54.781269 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:54.781277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:54.781285 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:11:54.781401 (+ 116us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:54.786039 (+ 4638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:54.786915 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:54.786928 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:54.786934 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:54.787129 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:11:54.787847 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:11:54.787854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:11:54.787860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:11:54.788372 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:54.788653 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:54.789199 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 01:11:54.789423 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 01:11:54.794049 21547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:54.803802 21547 catalog_manager.cc:1347] Generated new cluster ID: 76ee9dd2a70a495082235481aca7c865
I20260619 01:11:54.803941 21547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:54.821674 21547 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:54.823012 21547 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:54.829190 21547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4538a08ca14e4c9bbd95f19e7b510097: Generated new TSK 0
I20260619 01:11:54.830084 21547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:54.842993 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21459
I20260619 01:11:54.844362 21521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:11:54.938463 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 21459
W20260619 01:11:54.955953 21444 connection.cc:535] client connection to 127.0.0.1:34371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34371): Connection reset by peer (error 104)
2026-06-19T01:11:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:55 dist-test-slave-5p8w krb5kdc[21558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:55 dist-test-slave-5p8w krb5kdc[21558](info): set up 2 sockets
Jun 19 01:11:55 dist-test-slave-5p8w krb5kdc[21558](info): commencing operation
krb5kdc: starting...
W20260619 01:11:56.041702 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:56 dist-test-slave-5p8w krb5kdc[21558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:11:56.302208 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:56.645627 21571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:56.645998 21571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:56.646162 21571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:56.653580 21571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:56.653751 21571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:56.653838 21571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:56.653941 21571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:56.654023 21571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:56.664120 21571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:56.665611 21571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:56.667034 21571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:56.679384 21576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:56.680063 21579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:56.680065 21577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:56.680347 21571 server_base.cc:1034] running on GCE node
I20260619 01:11:56.681173 21571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:56.682646 21571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:56.683868 21571 hybrid_clock.cc:648] HybridClock initialized: now 1781831516683822 us; error 34 us; skew 500 ppm
I20260619 01:11:56.684222 21571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:56 dist-test-slave-5p8w krb5kdc[21558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:56.687651 21571 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:11:56.689168 21571 webserver.cc:460] Webserver started at http://127.0.0.1:34285/ using document root <none> and password file <none>
I20260619 01:11:56.689886 21571 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:56.690033 21571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:56.690340 21571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:56.693413 21571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "da27de18ed4a4998806570beec193a3b"
format_stamp: "Formatted at 2026-06-19 01:11:56 on dist-test-slave-5p8w"
server_key: "b3f69b52205dd9143b032025d5b18c06"
server_key_iv: "7ba2493078fc978008662a0583f1d396"
server_key_version: "encryptionkey@0"
I20260619 01:11:56.694209 21571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "da27de18ed4a4998806570beec193a3b"
format_stamp: "Formatted at 2026-06-19 01:11:56 on dist-test-slave-5p8w"
server_key: "b3f69b52205dd9143b032025d5b18c06"
server_key_iv: "7ba2493078fc978008662a0583f1d396"
server_key_version: "encryptionkey@0"
I20260619 01:11:56.699560 21571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:11:56.703112 21586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:56.704501 21571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:11:56.704701 21571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "da27de18ed4a4998806570beec193a3b"
format_stamp: "Formatted at 2026-06-19 01:11:56 on dist-test-slave-5p8w"
server_key: "b3f69b52205dd9143b032025d5b18c06"
server_key_iv: "7ba2493078fc978008662a0583f1d396"
server_key_version: "encryptionkey@0"
I20260619 01:11:56.704911 21571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:56.737002 21571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:56.741324 21571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:56.741619 21571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:56.760679 21571 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40465
I20260619 01:11:56.760710 21637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40465 every 8 connection(s)
I20260619 01:11:56.762181 21571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:56.766163 21638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:56.768061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 21571
I20260619 01:11:56.768271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:56.768802 388 external_mini_cluster.cc:1428] Setting key 99dcb1780a77f33e11290a0fff9ba62c
I20260619 01:11:56.780649 21638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b: Bootstrap starting.
I20260619 01:11:56.782574 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:56.770789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:55673 (local address 127.0.0.1:40465)
0619 01:11:56.771659 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:56.771672 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:56.771716 (+ 44us) server_negotiation.cc:407] Connection header received
0619 01:11:56.773450 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:56.773475 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:56.773921 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:56.774269 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:56.774816 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:56.776146 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:56.778120 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:56.778788 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:56.778923 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:56.780222 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:56.780230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:56.780244 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:56.780833 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:56.780914 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:56.780939 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:56.781057 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:56.781431 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 01:11:56.781623 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":416,"threads_started":1}
I20260619 01:11:56.783988 21638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:56.785073 21638 log.cc:824] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:56.786427 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:11:56.787869 21638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b: No bootstrap required, opened a new log
I20260619 01:11:56.797755 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:56.790795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:47175 (local address 127.0.0.1:40465)
0619 01:11:56.790938 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:56.790943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:56.791481 (+ 538us) server_negotiation.cc:407] Connection header received
0619 01:11:56.791663 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:56.791666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:56.791723 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:56.791799 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:56.792409 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:56.793185 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:56.795116 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:11:56.795575 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:11:56.795670 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:11:56.796904 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:56.796909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:56.796912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:56.796930 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:56.796982 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:56.796996 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:56.797074 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:56.797284 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 01:11:56.797372 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 01:11:56.797509 21638 raft_consensus.cc:348] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da27de18ed4a4998806570beec193a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260619 01:11:56.797996 21638 raft_consensus.cc:374] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:56.798126 21638 raft_consensus.cc:729] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da27de18ed4a4998806570beec193a3b, State: Initialized, Role: FOLLOWER
I20260619 01:11:56.798712 21638 consensus_queue.cc:260] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da27de18ed4a4998806570beec193a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260619 01:11:56.798990 21638 raft_consensus.cc:388] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:56.799154 21638 raft_consensus.cc:482] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:56.799328 21638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:56.801540 21638 raft_consensus.cc:504] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da27de18ed4a4998806570beec193a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260619 01:11:56.802042 21638 leader_election.cc:302] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da27de18ed4a4998806570beec193a3b; no voters:
I20260619 01:11:56.802630 21638 leader_election.cc:288] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:56.802783 21648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:56.803499 21648 raft_consensus.cc:686] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [term 1 LEADER]: Becoming Leader. State: Replica: da27de18ed4a4998806570beec193a3b, State: Running, Role: LEADER
I20260619 01:11:56.804031 21648 consensus_queue.cc:237] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da27de18ed4a4998806570beec193a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } }
I20260619 01:11:56.804694 21638 sys_catalog.cc:564] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:56.807407 21649 sys_catalog.cc:455] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da27de18ed4a4998806570beec193a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da27de18ed4a4998806570beec193a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } } }
I20260619 01:11:56.807514 21650 sys_catalog.cc:455] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader da27de18ed4a4998806570beec193a3b. Latest consensus state: current_term: 1 leader_uuid: "da27de18ed4a4998806570beec193a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da27de18ed4a4998806570beec193a3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40465 } } }
I20260619 01:11:56.807883 21649 sys_catalog.cc:458] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:56.807960 21650 sys_catalog.cc:458] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:56.809965 21656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:56.815060 21656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:56.824980 21656 catalog_manager.cc:1347] Generated new cluster ID: 9046e0a2fac340d7b860a29025a01ed4
I20260619 01:11:56.825132 21656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:56.834995 21656 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:56.836318 21656 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:56.843770 21656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da27de18ed4a4998806570beec193a3b: Generated new TSK 0
I20260619 01:11:56.844614 21656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:56.856375 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21571
W20260619 01:11:56.857507 21633 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.131:55673
W20260619 01:11:56.858292 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21571: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:11:56.858439 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 21571
W20260619 01:11:56.875379 21556 connection.cc:535] client connection to 127.0.0.1:40465 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40465): Connection reset by peer (error 104)
2026-06-19T01:11:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:56 dist-test-slave-5p8w krb5kdc[21669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:56 dist-test-slave-5p8w krb5kdc[21669](info): set up 2 sockets
Jun 19 01:11:56 dist-test-slave-5p8w krb5kdc[21669](info): commencing operation
krb5kdc: starting...
W20260619 01:11:57.981796 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:11:58 dist-test-slave-5p8w krb5kdc[21669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:11:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:11:58.238711 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:11:58.577890 21682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:11:58.578258 21682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:11:58.578423 21682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:11:58.585983 21682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:11:58.586169 21682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:11:58.586279 21682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:11:58.586385 21682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:11:58.586467 21682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:11:58.597043 21682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:11:58.598441 21682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:11:58.599802 21682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:11:58.612386 21687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:58.612349 21688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:11:58.612355 21690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:11:58.613547 21682 server_base.cc:1034] running on GCE node
I20260619 01:11:58.614388 21682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:11:58.616050 21682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:11:58.617337 21682 hybrid_clock.cc:648] HybridClock initialized: now 1781831518617153 us; error 144 us; skew 500 ppm
I20260619 01:11:58.617736 21682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:11:58 dist-test-slave-5p8w krb5kdc[21669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:11:58.626240 21682 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:11:58.627892 21682 webserver.cc:460] Webserver started at http://127.0.0.1:36111/ using document root <none> and password file <none>
I20260619 01:11:58.628648 21682 fs_manager.cc:359] Metadata directory not provided
I20260619 01:11:58.628762 21682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:11:58.629048 21682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:11:58.632462 21682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "b19f2bfc6cfd4baba661713ee5ea19ae"
format_stamp: "Formatted at 2026-06-19 01:11:58 on dist-test-slave-5p8w"
server_key: "6a9dd21fd0867dee10a7483477205e05"
server_key_iv: "f80329615e7ed7f1d4ec3fa5b8310ad6"
server_key_version: "encryptionkey@0"
I20260619 01:11:58.633235 21682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "b19f2bfc6cfd4baba661713ee5ea19ae"
format_stamp: "Formatted at 2026-06-19 01:11:58 on dist-test-slave-5p8w"
server_key: "6a9dd21fd0867dee10a7483477205e05"
server_key_iv: "f80329615e7ed7f1d4ec3fa5b8310ad6"
server_key_version: "encryptionkey@0"
I20260619 01:11:58.638772 21682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:11:58.642450 21697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:11:58.643884 21682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:11:58.644027 21682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "b19f2bfc6cfd4baba661713ee5ea19ae"
format_stamp: "Formatted at 2026-06-19 01:11:58 on dist-test-slave-5p8w"
server_key: "6a9dd21fd0867dee10a7483477205e05"
server_key_iv: "f80329615e7ed7f1d4ec3fa5b8310ad6"
server_key_version: "encryptionkey@0"
I20260619 01:11:58.644217 21682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:11:58.650134 21682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:11:58.654502 21682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:11:58.654758 21682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:11:58.673558 21682 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40897
I20260619 01:11:58.673599 21748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40897 every 8 connection(s)
I20260619 01:11:58.674983 21682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:11:58.678702 21749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:11:58.685262 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 21682
I20260619 01:11:58.685503 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:11:58.686057 388 external_mini_cluster.cc:1428] Setting key 40b7f835faac57c43a8d621e5d0a742f
I20260619 01:11:58.690462 21749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae: Bootstrap starting.
I20260619 01:11:58.693939 21749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae: Neither blocks nor log segments found. Creating new log.
I20260619 01:11:58.694240 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:58.688021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:43199 (local address 127.0.0.1:40897)
0619 01:11:58.688862 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:58.688875 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:58.688916 (+ 41us) server_negotiation.cc:407] Connection header received
0619 01:11:58.690772 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:58.690797 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:58.691158 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:58.691536 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:58.692592 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:58.692598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:58.692611 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:58.692627 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:58.692684 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:58.692703 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:58.692914 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:58.693045 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 01:11:58.693238 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":362,"threads_started":1}
I20260619 01:11:58.695299 21749 log.cc:824] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae: Log is configured to *not* fsync() on all Append() calls
I20260619 01:11:58.698107 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:11:58.698426 21749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae: No bootstrap required, opened a new log
I20260619 01:11:58.704778 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:11:58.702313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:39013 (local address 127.0.0.1:40897)
0619 01:11:58.702495 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 01:11:58.702506 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:11:58.702907 (+ 401us) server_negotiation.cc:407] Connection header received
0619 01:11:58.703038 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:11:58.703044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:11:58.703111 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:11:58.703209 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:11:58.704131 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:11:58.704137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:11:58.704142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:11:58.704160 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:11:58.704211 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:11:58.704235 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:11:58.704359 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 01:11:58.704405 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 01:11:58.704466 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 01:11:58.706916 21749 raft_consensus.cc:348] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19f2bfc6cfd4baba661713ee5ea19ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40897 } }
I20260619 01:11:58.707265 21749 raft_consensus.cc:374] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:11:58.707342 21749 raft_consensus.cc:729] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19f2bfc6cfd4baba661713ee5ea19ae, State: Initialized, Role: FOLLOWER
I20260619 01:11:58.707816 21749 consensus_queue.cc:260] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19f2bfc6cfd4baba661713ee5ea19ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40897 } }
I20260619 01:11:58.708014 21749 raft_consensus.cc:388] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:11:58.708107 21749 raft_consensus.cc:482] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:11:58.708238 21749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:11:58.710450 21749 raft_consensus.cc:504] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19f2bfc6cfd4baba661713ee5ea19ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40897 } }
I20260619 01:11:58.710958 21749 leader_election.cc:302] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b19f2bfc6cfd4baba661713ee5ea19ae; no voters:
I20260619 01:11:58.711529 21749 leader_election.cc:288] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:11:58.711664 21759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:11:58.712371 21759 raft_consensus.cc:686] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [term 1 LEADER]: Becoming Leader. State: Replica: b19f2bfc6cfd4baba661713ee5ea19ae, State: Running, Role: LEADER
I20260619 01:11:58.712863 21759 consensus_queue.cc:237] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19f2bfc6cfd4baba661713ee5ea19ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40897 } }
I20260619 01:11:58.713685 21749 sys_catalog.cc:564] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:11:58.715512 21760 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b19f2bfc6cfd4baba661713ee5ea19ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19f2bfc6cfd4baba661713ee5ea19ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40897 } } }
I20260619 01:11:58.716048 21760 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:58.715886 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b19f2bfc6cfd4baba661713ee5ea19ae. Latest consensus state: current_term: 1 leader_uuid: "b19f2bfc6cfd4baba661713ee5ea19ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19f2bfc6cfd4baba661713ee5ea19ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40897 } } }
I20260619 01:11:58.716200 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae [sys.catalog]: This master's current role is: LEADER
I20260619 01:11:58.716928 21764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:11:58.722889 21764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:11:58.731357 21764 catalog_manager.cc:1347] Generated new cluster ID: c4e3098607ae45aab7ce42f37117e33e
I20260619 01:11:58.731475 21764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:11:58.732993 21776 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 01:11:58.749190 21764 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:11:58.749687 21764 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:11:58.760649 21764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b19f2bfc6cfd4baba661713ee5ea19ae: Generated new TSK 0
I20260619 01:11:58.761575 21764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:11:58.770893 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21682
W20260619 01:11:58.772879 21744 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.131:43199
W20260619 01:11:58.773691 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21682: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:11:58.773833 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 21682
2026-06-19T01:11:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1915 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:11:58 dist-test-slave-5p8w krb5kdc[21780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:11:58 dist-test-slave-5p8w krb5kdc[21780](info): set up 2 sockets
Jun 19 01:11:58 dist-test-slave-5p8w krb5kdc[21780](info): commencing operation
krb5kdc: starting...
W20260619 01:11:59.873733 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:12:00 dist-test-slave-5p8w krb5kdc[21780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:12:00.138537 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:12:00.474144 21793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:00.474452 21793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:00.474635 21793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:00.482002 21793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:00.482141 21793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:00.482244 21793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:00.482347 21793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:00.482420 21793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:00.492581 21793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:00.494021 21793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:00.495402 21793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:00.507555 21798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:00.507670 21801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:00.507683 21799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:00.508708 21793 server_base.cc:1034] running on GCE node
I20260619 01:12:00.509857 21793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:00.511865 21793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:00.513053 21793 hybrid_clock.cc:648] HybridClock initialized: now 1781831520512977 us; error 72 us; skew 500 ppm
I20260619 01:12:00.513484 21793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:00 dist-test-slave-5p8w krb5kdc[21780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:00.516561 21793 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:12:00.518179 21793 webserver.cc:460] Webserver started at http://127.0.0.1:38865/ using document root <none> and password file <none>
I20260619 01:12:00.518920 21793 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:00.519068 21793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:00.519390 21793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:00.522823 21793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "8adcc36366fc4829a8aab3649cbcaccf"
format_stamp: "Formatted at 2026-06-19 01:12:00 on dist-test-slave-5p8w"
server_key: "b2fd8b8d4562564c10684a82f31ca209"
server_key_iv: "835bf9d594a0206f31ce97341c581c01"
server_key_version: "encryptionkey@0"
I20260619 01:12:00.523687 21793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "8adcc36366fc4829a8aab3649cbcaccf"
format_stamp: "Formatted at 2026-06-19 01:12:00 on dist-test-slave-5p8w"
server_key: "b2fd8b8d4562564c10684a82f31ca209"
server_key_iv: "835bf9d594a0206f31ce97341c581c01"
server_key_version: "encryptionkey@0"
I20260619 01:12:00.529405 21793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 01:12:00.533061 21808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:00.534597 21793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:12:00.534890 21793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "8adcc36366fc4829a8aab3649cbcaccf"
format_stamp: "Formatted at 2026-06-19 01:12:00 on dist-test-slave-5p8w"
server_key: "b2fd8b8d4562564c10684a82f31ca209"
server_key_iv: "835bf9d594a0206f31ce97341c581c01"
server_key_version: "encryptionkey@0"
I20260619 01:12:00.535089 21793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:00.558106 21793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:00.562453 21793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:00.562767 21793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:00.582149 21793 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39537
I20260619 01:12:00.582180 21859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39537 every 8 connection(s)
I20260619 01:12:00.583599 21793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:12:00.584686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 21793
I20260619 01:12:00.584870 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:12:00.585413 388 external_mini_cluster.cc:1428] Setting key 98d7a1a76f487c663a4260a8d9368823
I20260619 01:12:00.588336 21860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:12:00 dist-test-slave-5p8w krb5kdc[21780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 01:12:00.602312 21860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf: Bootstrap starting.
I20260619 01:12:00.605984 21860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:00.607105 21860 log.cc:824] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:00.609941 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:00.587269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:60697 (local address 127.0.0.1:39537)
0619 01:12:00.588101 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:00.588113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:00.588157 (+ 44us) server_negotiation.cc:407] Connection header received
0619 01:12:00.590009 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:00.590033 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:00.590461 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:00.590827 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:00.592703 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:00.594167 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:00.596001 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:00.596667 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:00.596803 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:00.601936 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:00.601946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:00.601964 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:00.602007 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:00.605170 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:00.606086 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:00.606094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:00.606121 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:00.606334 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:00.606810 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:00.606820 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:00.606822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:00.607398 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:00.607750 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:00.608298 (+ 548us) server_negotiation.cc:299] Negotiation successful
0619 01:12:00.608831 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":395,"threads_started":1}
I20260619 01:12:00.610272 21860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf: No bootstrap required, opened a new log
I20260619 01:12:00.614125 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:12:00.618534 21860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adcc36366fc4829a8aab3649cbcaccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260619 01:12:00.619053 21860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:12:00.619195 21860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8adcc36366fc4829a8aab3649cbcaccf, State: Initialized, Role: FOLLOWER
I20260619 01:12:00.619922 21860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adcc36366fc4829a8aab3649cbcaccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260619 01:12:00.620249 21860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:12:00.620393 21860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:12:00.620594 21860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:00.623749 21860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adcc36366fc4829a8aab3649cbcaccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260619 01:12:00.624401 21860 leader_election.cc:302] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8adcc36366fc4829a8aab3649cbcaccf; no voters:
I20260619 01:12:00.625195 21860 leader_election.cc:288] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:12:00.625339 21870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:12:00.626555 21870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [term 1 LEADER]: Becoming Leader. State: Replica: 8adcc36366fc4829a8aab3649cbcaccf, State: Running, Role: LEADER
I20260619 01:12:00.627206 21870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adcc36366fc4829a8aab3649cbcaccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260619 01:12:00.627786 21860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:00.631248 21872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8adcc36366fc4829a8aab3649cbcaccf. Latest consensus state: current_term: 1 leader_uuid: "8adcc36366fc4829a8aab3649cbcaccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adcc36366fc4829a8aab3649cbcaccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } } }
I20260619 01:12:00.631795 21872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:00.632361 21871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8adcc36366fc4829a8aab3649cbcaccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8adcc36366fc4829a8aab3649cbcaccf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } } }
I20260619 01:12:00.632694 21871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:00.638952 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:00.618669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:50583 (local address 127.0.0.1:39537)
0619 01:12:00.618874 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:00.618882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:00.619548 (+ 666us) server_negotiation.cc:407] Connection header received
0619 01:12:00.619755 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:00.619759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:00.619829 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:00.619917 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:00.621912 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:00.622853 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:00.624685 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:00.625149 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:00.625344 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:00.628305 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:00.628310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:00.628314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:00.628347 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:00.632730 (+ 4383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:00.634492 (+ 1762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:00.634501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:00.634507 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:00.634810 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:00.635563 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:00.635571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:00.635577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:00.636076 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:00.636490 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:00.637971 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0619 01:12:00.638245 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:12:00.642851 21877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:12:00.648339 21877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:12:00.658246 21877 catalog_manager.cc:1347] Generated new cluster ID: c76464883ac24186b721e14716fa1935
I20260619 01:12:00.658380 21877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:12:00.668763 21877 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:12:00.670173 21877 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:12:00.680389 21877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8adcc36366fc4829a8aab3649cbcaccf: Generated new TSK 0
I20260619 01:12:00.681380 21877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:12:00.688935 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21793
I20260619 01:12:00.690227 21855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:12:00.780583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 21793
W20260619 01:12:00.797835 21778 connection.cc:535] client connection to 127.0.0.1:39537 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39537): Connection reset by peer (error 104)
2026-06-19T01:12:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:12:00 dist-test-slave-5p8w krb5kdc[21893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:12:00 dist-test-slave-5p8w krb5kdc[21893](info): set up 2 sockets
Jun 19 01:12:00 dist-test-slave-5p8w krb5kdc[21893](info): commencing operation
krb5kdc: starting...
W20260619 01:12:01.905721 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:12:02 dist-test-slave-5p8w krb5kdc[21893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:12:02.166622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:12:02.515498 21906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:02.516008 21906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:02.516192 21906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:02.525066 21906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:02.525243 21906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:02.525384 21906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:02.525518 21906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:02.525614 21906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:02.536244 21906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:02.537777 21906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:02.539144 21906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:02.551968 21912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:02.551877 21914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:02.552151 21911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:02.552673 21906 server_base.cc:1034] running on GCE node
I20260619 01:12:02.553650 21906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:02.555208 21906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:02.556443 21906 hybrid_clock.cc:648] HybridClock initialized: now 1781831522556380 us; error 62 us; skew 500 ppm
I20260619 01:12:02.556871 21906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:02 dist-test-slave-5p8w krb5kdc[21893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:02.560130 21906 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:12:02.561738 21906 webserver.cc:460] Webserver started at http://127.0.0.1:42479/ using document root <none> and password file <none>
I20260619 01:12:02.562451 21906 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:02.562605 21906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:02.562919 21906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:02.566023 21906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "6055ce43cd284cf8a07b8fa15ce523ef"
format_stamp: "Formatted at 2026-06-19 01:12:02 on dist-test-slave-5p8w"
server_key: "9993640b6b29ea5429b6b89c16a4d498"
server_key_iv: "3bd89d4d3c5545bd33f432560a2e9201"
server_key_version: "encryptionkey@0"
I20260619 01:12:02.566843 21906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "6055ce43cd284cf8a07b8fa15ce523ef"
format_stamp: "Formatted at 2026-06-19 01:12:02 on dist-test-slave-5p8w"
server_key: "9993640b6b29ea5429b6b89c16a4d498"
server_key_iv: "3bd89d4d3c5545bd33f432560a2e9201"
server_key_version: "encryptionkey@0"
I20260619 01:12:02.572153 21906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:12:02.575604 21921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:02.576933 21906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:12:02.577116 21906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "6055ce43cd284cf8a07b8fa15ce523ef"
format_stamp: "Formatted at 2026-06-19 01:12:02 on dist-test-slave-5p8w"
server_key: "9993640b6b29ea5429b6b89c16a4d498"
server_key_iv: "3bd89d4d3c5545bd33f432560a2e9201"
server_key_version: "encryptionkey@0"
I20260619 01:12:02.577344 21906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:02.603006 21906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:02.607031 21906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:02.607327 21906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:02.625901 21906 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46601
I20260619 01:12:02.625933 21972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46601 every 8 connection(s)
I20260619 01:12:02.627401 21906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:12:02.631206 21973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:02.633266 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 21906
I20260619 01:12:02.633497 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:12:02.633917 388 external_mini_cluster.cc:1428] Setting key b3b94e214103c07e039c92b63c8efeb2
I20260619 01:12:02.645787 21973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef: Bootstrap starting.
I20260619 01:12:02.648408 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:02.635886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:36651 (local address 127.0.0.1:46601)
0619 01:12:02.636690 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:02.636708 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:02.636770 (+ 62us) server_negotiation.cc:407] Connection header received
0619 01:12:02.638533 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:02.638563 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:02.639118 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:02.639526 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:02.640019 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:02.641969 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:02.643304 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:02.644315 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:02.644508 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:02.645646 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:02.645656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:02.645674 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:02.646390 (+ 716us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:02.646480 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:02.646511 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:02.646673 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:02.646913 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 01:12:02.647080 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":316,"threads_started":1}
I20260619 01:12:02.649842 21973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:02.651036 21973 log.cc:824] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:02.652460 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:12:02.654417 21973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef: No bootstrap required, opened a new log
I20260619 01:12:02.664225 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:02.657648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:37197 (local address 127.0.0.1:46601)
0619 01:12:02.657854 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:02.657861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:02.658213 (+ 352us) server_negotiation.cc:407] Connection header received
0619 01:12:02.658347 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:02.658352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:02.658439 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:02.658559 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:02.658996 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:02.660136 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:02.661533 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:02.662233 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:02.662375 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:02.663457 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:02.663463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:02.663469 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:02.663492 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:02.663560 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:02.663581 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:02.663711 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:02.663858 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 01:12:02.663915 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 01:12:02.664887 21973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6055ce43cd284cf8a07b8fa15ce523ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } }
I20260619 01:12:02.665468 21973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:12:02.665584 21973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6055ce43cd284cf8a07b8fa15ce523ef, State: Initialized, Role: FOLLOWER
I20260619 01:12:02.666254 21973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6055ce43cd284cf8a07b8fa15ce523ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } }
I20260619 01:12:02.666503 21973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:12:02.666659 21973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:12:02.666844 21973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:02.669283 21973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6055ce43cd284cf8a07b8fa15ce523ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } }
I20260619 01:12:02.669854 21973 leader_election.cc:302] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6055ce43cd284cf8a07b8fa15ce523ef; no voters:
I20260619 01:12:02.670445 21973 leader_election.cc:288] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:12:02.670596 21983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:12:02.671231 21983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [term 1 LEADER]: Becoming Leader. State: Replica: 6055ce43cd284cf8a07b8fa15ce523ef, State: Running, Role: LEADER
I20260619 01:12:02.671865 21983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6055ce43cd284cf8a07b8fa15ce523ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } }
I20260619 01:12:02.673722 21983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6055ce43cd284cf8a07b8fa15ce523ef. Latest consensus state: current_term: 1 leader_uuid: "6055ce43cd284cf8a07b8fa15ce523ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6055ce43cd284cf8a07b8fa15ce523ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } } }
I20260619 01:12:02.673996 21983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:02.674665 21985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6055ce43cd284cf8a07b8fa15ce523ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6055ce43cd284cf8a07b8fa15ce523ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } } }
I20260619 01:12:02.675034 21985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:02.678947 21973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:02.679157 21987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:12:02.685889 21987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:12:02.694773 21987 catalog_manager.cc:1347] Generated new cluster ID: c540adf60e3f4ad48cb8d5a6242dc2d1
I20260619 01:12:02.694909 21987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:12:02.708503 21987 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:12:02.710480 21987 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:12:02.725828 21987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6055ce43cd284cf8a07b8fa15ce523ef: Generated new TSK 0
I20260619 01:12:02.726781 21987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:12:02.742741 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21906
W20260619 01:12:02.744217 21968 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.131:36651
W20260619 01:12:02.745038 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 21906: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:12:02.745203 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 21906
W20260619 01:12:02.762542 21891 connection.cc:535] client connection to 127.0.0.1:46601 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46601): Connection reset by peer (error 104)
2026-06-19T01:12:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:12:02 dist-test-slave-5p8w krb5kdc[22004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:12:02 dist-test-slave-5p8w krb5kdc[22004](info): set up 2 sockets
Jun 19 01:12:02 dist-test-slave-5p8w krb5kdc[22004](info): commencing operation
krb5kdc: starting...
W20260619 01:12:03.853884 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:12:04 dist-test-slave-5p8w krb5kdc[22004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:12:04.108904 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:12:04.455268 22017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:04.455579 22017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:04.455672 22017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:04.463140 22017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:04.463276 22017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:04.463330 22017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:04.463389 22017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:04.463433 22017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:04.473623 22017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:04.474983 22017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:04.476264 22017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:04.488596 22022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:04.488545 22025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:04.488667 22023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:04.489423 22017 server_base.cc:1034] running on GCE node
I20260619 01:12:04.490181 22017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:04.491580 22017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:04.492790 22017 hybrid_clock.cc:648] HybridClock initialized: now 1781831524492750 us; error 31 us; skew 500 ppm
I20260619 01:12:04.493170 22017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:04 dist-test-slave-5p8w krb5kdc[22004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:04.500355 22017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:12:04.502022 22017 webserver.cc:460] Webserver started at http://127.0.0.1:45279/ using document root <none> and password file <none>
I20260619 01:12:04.502748 22017 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:04.502882 22017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:04.503209 22017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:04.506367 22017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "f36cd8063d9d48b1a0b5e46eebe4bdd0"
format_stamp: "Formatted at 2026-06-19 01:12:04 on dist-test-slave-5p8w"
server_key: "2228a3f9f62b30c8879a0c2b2b08a713"
server_key_iv: "dbb1d9066ea652d4531af8a9f8f91eab"
server_key_version: "encryptionkey@0"
I20260619 01:12:04.507124 22017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "f36cd8063d9d48b1a0b5e46eebe4bdd0"
format_stamp: "Formatted at 2026-06-19 01:12:04 on dist-test-slave-5p8w"
server_key: "2228a3f9f62b30c8879a0c2b2b08a713"
server_key_iv: "dbb1d9066ea652d4531af8a9f8f91eab"
server_key_version: "encryptionkey@0"
I20260619 01:12:04.512410 22017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:12:04.515960 22032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:04.517413 22017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:12:04.517606 22017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "f36cd8063d9d48b1a0b5e46eebe4bdd0"
format_stamp: "Formatted at 2026-06-19 01:12:04 on dist-test-slave-5p8w"
server_key: "2228a3f9f62b30c8879a0c2b2b08a713"
server_key_iv: "dbb1d9066ea652d4531af8a9f8f91eab"
server_key_version: "encryptionkey@0"
I20260619 01:12:04.517843 22017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:04.524039 22017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:04.529284 22017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:04.529662 22017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:04.547245 22017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46267
I20260619 01:12:04.547276 22083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46267 every 8 connection(s)
I20260619 01:12:04.548739 22017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:12:04.552580 22084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:04.555670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 22017
I20260619 01:12:04.555866 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:12:04.556417 388 external_mini_cluster.cc:1428] Setting key 080289d3dc011ae2adb0260101228d39
I20260619 01:12:04.565647 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:04.558370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:35601 (local address 127.0.0.1:46267)
0619 01:12:04.559174 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:04.559202 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:04.559260 (+ 58us) server_negotiation.cc:407] Connection header received
0619 01:12:04.561628 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:04.561658 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:04.562140 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:04.562616 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:04.563631 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:04.563640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:04.563657 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:04.563674 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:04.563744 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:04.563774 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:04.564014 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:04.564151 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 01:12:04.564383 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":319,"threads_started":1}
I20260619 01:12:04.566726 22084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0: Bootstrap starting.
I20260619 01:12:04.569766 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:12:04.570649 22084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:04.571920 22084 log.cc:824] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:04.575918 22084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0: No bootstrap required, opened a new log
I20260619 01:12:04.580441 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:04.577956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:59391 (local address 127.0.0.1:46267)
0619 01:12:04.578161 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:04.578168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:04.578575 (+ 407us) server_negotiation.cc:407] Connection header received
0619 01:12:04.578742 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:04.578748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:04.578813 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:04.578920 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:04.579814 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:04.579820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:04.579825 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:04.579843 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:04.579886 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:04.579904 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:04.580032 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:04.580082 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 01:12:04.580142 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:12:04.584944 22084 raft_consensus.cc:348] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cd8063d9d48b1a0b5e46eebe4bdd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260619 01:12:04.585422 22084 raft_consensus.cc:374] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:12:04.585505 22084 raft_consensus.cc:729] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36cd8063d9d48b1a0b5e46eebe4bdd0, State: Initialized, Role: FOLLOWER
I20260619 01:12:04.586031 22084 consensus_queue.cc:260] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cd8063d9d48b1a0b5e46eebe4bdd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260619 01:12:04.586256 22084 raft_consensus.cc:388] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:12:04.586429 22084 raft_consensus.cc:482] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:12:04.586609 22084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:04.589099 22084 raft_consensus.cc:504] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cd8063d9d48b1a0b5e46eebe4bdd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260619 01:12:04.589731 22084 leader_election.cc:302] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f36cd8063d9d48b1a0b5e46eebe4bdd0; no voters:
I20260619 01:12:04.590335 22084 leader_election.cc:288] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:12:04.590521 22094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:12:04.591948 22094 raft_consensus.cc:686] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [term 1 LEADER]: Becoming Leader. State: Replica: f36cd8063d9d48b1a0b5e46eebe4bdd0, State: Running, Role: LEADER
I20260619 01:12:04.592636 22084 sys_catalog.cc:564] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:04.592495 22094 consensus_queue.cc:237] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cd8063d9d48b1a0b5e46eebe4bdd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260619 01:12:04.595818 22096 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f36cd8063d9d48b1a0b5e46eebe4bdd0. Latest consensus state: current_term: 1 leader_uuid: "f36cd8063d9d48b1a0b5e46eebe4bdd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cd8063d9d48b1a0b5e46eebe4bdd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } } }
I20260619 01:12:04.596256 22096 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:04.595882 22095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f36cd8063d9d48b1a0b5e46eebe4bdd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cd8063d9d48b1a0b5e46eebe4bdd0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } } }
I20260619 01:12:04.596865 22095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:04.599048 22104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:12:04.603266 22104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:12:04.611299 22104 catalog_manager.cc:1347] Generated new cluster ID: bb34826df056449c979f5c7da3f1c1a9
I20260619 01:12:04.611419 22104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:12:04.621735 22104 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:12:04.622234 22104 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:12:04.640836 22104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f36cd8063d9d48b1a0b5e46eebe4bdd0: Generated new TSK 0
I20260619 01:12:04.641901 22104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:12:04.656165 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22017
W20260619 01:12:04.657591 22079 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.131:35601
W20260619 01:12:04.658324 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22017: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:12:04.658468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 22017
2026-06-19T01:12:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:12:04 dist-test-slave-5p8w krb5kdc[22115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:12:04 dist-test-slave-5p8w krb5kdc[22115](info): set up 2 sockets
Jun 19 01:12:04 dist-test-slave-5p8w krb5kdc[22115](info): commencing operation
krb5kdc: starting...
W20260619 01:12:05.769716 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:12:05 dist-test-slave-5p8w krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:12:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:12:06.022432 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37817
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:12:06.358847 22128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:06.359232 22128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:06.359402 22128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:06.366890 22128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:06.367057 22128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:06.367157 22128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:06.367276 22128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:06.367358 22128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:06.377722 22128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:06.379204 22128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:06.380605 22128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:06.392949 22133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:06.392979 22136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:06.393016 22134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:06.393988 22128 server_base.cc:1034] running on GCE node
I20260619 01:12:06.394886 22128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:06.396394 22128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:06.397650 22128 hybrid_clock.cc:648] HybridClock initialized: now 1781831526397577 us; error 64 us; skew 500 ppm
I20260619 01:12:06.398030 22128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:06 dist-test-slave-5p8w krb5kdc[22115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:06.401121 22128 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:12:06.402761 22128 webserver.cc:460] Webserver started at http://127.0.97.62:32851/ using document root <none> and password file <none>
I20260619 01:12:06.403502 22128 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:06.403661 22128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:06.403977 22128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:06.407212 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "8bece3bcd8304334ad9878b2f0399858"
format_stamp: "Formatted at 2026-06-19 01:12:06 on dist-test-slave-5p8w"
server_key: "a784723aabe13b6018140b92f4db79e5"
server_key_iv: "949e1ec26f5a47ad12771db612315810"
server_key_version: "encryptionkey@0"
I20260619 01:12:06.408012 22128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "8bece3bcd8304334ad9878b2f0399858"
format_stamp: "Formatted at 2026-06-19 01:12:06 on dist-test-slave-5p8w"
server_key: "a784723aabe13b6018140b92f4db79e5"
server_key_iv: "949e1ec26f5a47ad12771db612315810"
server_key_version: "encryptionkey@0"
I20260619 01:12:06.413516 22128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 01:12:06.417001 22143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:06.418442 22128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:12:06.418644 22128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "8bece3bcd8304334ad9878b2f0399858"
format_stamp: "Formatted at 2026-06-19 01:12:06 on dist-test-slave-5p8w"
server_key: "a784723aabe13b6018140b92f4db79e5"
server_key_iv: "949e1ec26f5a47ad12771db612315810"
server_key_version: "encryptionkey@0"
I20260619 01:12:06.418856 22128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:06.439886 22128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:06.444418 22128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:06.444737 22128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:06.462896 22128 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37817
I20260619 01:12:06.462930 22194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37817 every 8 connection(s)
I20260619 01:12:06.464280 22128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:12:06.468078 22195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:06.468698 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 22128
I20260619 01:12:06.468875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:12:06.469488 388 external_mini_cluster.cc:1428] Setting key 8dae581081cb114a323e21b8def153cf
I20260619 01:12:06.482443 22195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858: Bootstrap starting.
Jun 19 01:12:06 dist-test-slave-5p8w krb5kdc[22115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:12:06.486289 22195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:06.487717 22195 log.cc:824] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:06.491140 22195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858: No bootstrap required, opened a new log
I20260619 01:12:06.494597 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:06.471345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47663 (local address 127.0.97.62:37817)
0619 01:12:06.472166 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:06.472178 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:06.472246 (+ 68us) server_negotiation.cc:407] Connection header received
0619 01:12:06.474491 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:06.474519 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:06.474961 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:06.475314 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:06.477018 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:06.478378 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:06.480311 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:06.480962 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:06.481086 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:06.485949 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:06.485961 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:06.485980 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:06.486028 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:06.489834 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:06.490695 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:06.490703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:06.490712 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:06.490910 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:06.491344 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:06.491349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:06.491352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:06.492050 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:06.492370 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:06.492732 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 01:12:06.493092 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":391,"threads_started":1}
I20260619 01:12:06.498721 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:12:06.500598 22195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bece3bcd8304334ad9878b2f0399858" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37817 } }
I20260619 01:12:06.500953 22195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:12:06.501075 22195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bece3bcd8304334ad9878b2f0399858, State: Initialized, Role: FOLLOWER
I20260619 01:12:06.501806 22195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bece3bcd8304334ad9878b2f0399858" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37817 } }
I20260619 01:12:06.502084 22195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:12:06.502218 22195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:12:06.502403 22195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:06.505044 22195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bece3bcd8304334ad9878b2f0399858" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37817 } }
I20260619 01:12:06.505599 22195 leader_election.cc:302] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bece3bcd8304334ad9878b2f0399858; no voters:
I20260619 01:12:06.506328 22195 leader_election.cc:288] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:12:06.506431 22205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:12:06.507694 22205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [term 1 LEADER]: Becoming Leader. State: Replica: 8bece3bcd8304334ad9878b2f0399858, State: Running, Role: LEADER
I20260619 01:12:06.508355 22205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bece3bcd8304334ad9878b2f0399858" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37817 } }
I20260619 01:12:06.508726 22195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:06.511317 22207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bece3bcd8304334ad9878b2f0399858. Latest consensus state: current_term: 1 leader_uuid: "8bece3bcd8304334ad9878b2f0399858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bece3bcd8304334ad9878b2f0399858" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37817 } } }
I20260619 01:12:06.511740 22207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:06.512403 22206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bece3bcd8304334ad9878b2f0399858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bece3bcd8304334ad9878b2f0399858" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37817 } } }
I20260619 01:12:06.512738 22206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:06.519229 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:06.503124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55965 (local address 127.0.97.62:37817)
0619 01:12:06.503282 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:06.503286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:06.503796 (+ 510us) server_negotiation.cc:407] Connection header received
0619 01:12:06.504000 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:06.504011 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:06.504078 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:06.504198 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:06.505722 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:06.506737 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:06.508494 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:06.509130 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:06.509275 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:06.512446 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:06.512454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:06.512459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:06.512500 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:06.516140 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:06.516876 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:06.516881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:06.516885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:06.517022 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:06.517708 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:06.517711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:06.517713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:06.517991 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:06.518272 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:06.518686 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 01:12:06.518851 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":52}
I20260619 01:12:06.522383 22212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:12:06.527501 22212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:12:06.537135 22212 catalog_manager.cc:1347] Generated new cluster ID: 08b0b6471bcc4d5f9e543a6cce0fba8d
I20260619 01:12:06.537256 22212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:12:06.559268 22212 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:12:06.561201 22212 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:12:06.569361 22212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bece3bcd8304334ad9878b2f0399858: Generated new TSK 0
I20260619 01:12:06.570393 22212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:12:06.580452 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22128
I20260619 01:12:06.582060 22190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:12:06.673245 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 22128
W20260619 01:12:06.690989 22113 connection.cc:535] client connection to 127.0.97.62:37817 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:37817): Connection reset by peer (error 104)
2026-06-19T01:12:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2015 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:12:06 dist-test-slave-5p8w krb5kdc[22227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:12:06 dist-test-slave-5p8w krb5kdc[22227](info): set up 2 sockets
Jun 19 01:12:06 dist-test-slave-5p8w krb5kdc[22227](info): commencing operation
krb5kdc: starting...
W20260619 01:12:07.777861 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:12:07 dist-test-slave-5p8w krb5kdc[22227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:12:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:12:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:12:08.037998 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41275
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:12:08.372813 22240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:08.373113 22240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:08.373284 22240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:08.380502 22240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:08.380611 22240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:08.380703 22240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:08.380790 22240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:08.380874 22240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:08.390719 22240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:08.392021 22240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:08.393353 22240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:08.404886 22245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:08.404948 22246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:08.404855 22248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:08.405532 22240 server_base.cc:1034] running on GCE node
I20260619 01:12:08.406304 22240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:08.407707 22240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:08.408946 22240 hybrid_clock.cc:648] HybridClock initialized: now 1781831528408905 us; error 49 us; skew 500 ppm
I20260619 01:12:08.409360 22240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:08 dist-test-slave-5p8w krb5kdc[22227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:08.412420 22240 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:12:08.413937 22240 webserver.cc:460] Webserver started at http://127.0.97.62:43283/ using document root <none> and password file <none>
I20260619 01:12:08.414621 22240 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:08.414768 22240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:08.415062 22240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:08.418130 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "c0bc988709cd46ac96ad00caa7b4d254"
format_stamp: "Formatted at 2026-06-19 01:12:08 on dist-test-slave-5p8w"
server_key: "b57245ce2e90105f4dade1b17d014223"
server_key_iv: "90249c626f0a546b40fec77dd00cd2c4"
server_key_version: "encryptionkey@0"
I20260619 01:12:08.418874 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "c0bc988709cd46ac96ad00caa7b4d254"
format_stamp: "Formatted at 2026-06-19 01:12:08 on dist-test-slave-5p8w"
server_key: "b57245ce2e90105f4dade1b17d014223"
server_key_iv: "90249c626f0a546b40fec77dd00cd2c4"
server_key_version: "encryptionkey@0"
I20260619 01:12:08.424189 22240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:12:08.427656 22255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:08.429075 22240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:12:08.429270 22240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "c0bc988709cd46ac96ad00caa7b4d254"
format_stamp: "Formatted at 2026-06-19 01:12:08 on dist-test-slave-5p8w"
server_key: "b57245ce2e90105f4dade1b17d014223"
server_key_iv: "90249c626f0a546b40fec77dd00cd2c4"
server_key_version: "encryptionkey@0"
I20260619 01:12:08.429531 22240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:08.442027 22240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:08.445919 22240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:08.446142 22240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:08.464318 22240 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41275
I20260619 01:12:08.464339 22306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41275 every 8 connection(s)
I20260619 01:12:08.465826 22240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:12:08.469615 22307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:08.473004 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 22240
I20260619 01:12:08.473176 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:12:08.473718 388 external_mini_cluster.cc:1428] Setting key 9f586fe404ba3a756787cb9b572b6809
I20260619 01:12:08.482263 22307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254: Bootstrap starting.
I20260619 01:12:08.486075 22307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:08.487267 22307 log.cc:824] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:08.487962 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:08.475640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40059 (local address 127.0.97.62:41275)
0619 01:12:08.476478 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:08.476491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:08.476534 (+ 43us) server_negotiation.cc:407] Connection header received
0619 01:12:08.478304 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:08.478330 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:08.478799 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:08.479170 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:08.479778 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:08.481090 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:08.483015 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:08.483798 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:08.483966 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:08.485399 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:08.485408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:08.485423 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:08.485988 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:08.486074 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:08.486098 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:08.486287 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:08.486612 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 01:12:08.486786 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":405,"threads_started":1}
I20260619 01:12:08.490590 22307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254: No bootstrap required, opened a new log
I20260619 01:12:08.491998 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:12:08.499358 22307 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc988709cd46ac96ad00caa7b4d254" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41275 } }
I20260619 01:12:08.499871 22307 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:12:08.499976 22307 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bc988709cd46ac96ad00caa7b4d254, State: Initialized, Role: FOLLOWER
I20260619 01:12:08.500658 22307 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc988709cd46ac96ad00caa7b4d254" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41275 } }
I20260619 01:12:08.500943 22307 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:12:08.501113 22307 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:12:08.501340 22307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:08.503849 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:08.496616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57463 (local address 127.0.97.62:41275)
0619 01:12:08.496980 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:08.497019 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:08.497532 (+ 513us) server_negotiation.cc:407] Connection header received
0619 01:12:08.497740 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:08.497744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:08.497807 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:08.497885 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:08.498441 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:08.499172 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:08.501065 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:08.501684 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:08.501780 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:08.503047 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:08.503052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:08.503056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:08.503075 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:08.503114 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:08.503132 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:08.503226 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:08.503424 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 01:12:08.503469 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 01:12:08.504354 22307 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc988709cd46ac96ad00caa7b4d254" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41275 } }
I20260619 01:12:08.504865 22307 leader_election.cc:302] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0bc988709cd46ac96ad00caa7b4d254; no voters:
I20260619 01:12:08.505554 22307 leader_election.cc:288] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:12:08.505676 22317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:12:08.506333 22317 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [term 1 LEADER]: Becoming Leader. State: Replica: c0bc988709cd46ac96ad00caa7b4d254, State: Running, Role: LEADER
I20260619 01:12:08.506961 22317 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc988709cd46ac96ad00caa7b4d254" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41275 } }
I20260619 01:12:08.507880 22307 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:08.510035 22318 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0bc988709cd46ac96ad00caa7b4d254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc988709cd46ac96ad00caa7b4d254" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41275 } } }
I20260619 01:12:08.510612 22318 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:08.510481 22319 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0bc988709cd46ac96ad00caa7b4d254. Latest consensus state: current_term: 1 leader_uuid: "c0bc988709cd46ac96ad00caa7b4d254" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bc988709cd46ac96ad00caa7b4d254" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41275 } } }
I20260619 01:12:08.510779 22319 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:08.513064 22325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:12:08.517663 22325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:12:08.526743 22325 catalog_manager.cc:1347] Generated new cluster ID: 47aa3fb27fa24aef9bf76324cd743b18
I20260619 01:12:08.526875 22325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:12:08.551570 22325 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:12:08.553406 22325 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:12:08.559928 22325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0bc988709cd46ac96ad00caa7b4d254: Generated new TSK 0
I20260619 01:12:08.560868 22325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:12:08.571014 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22240
W20260619 01:12:08.572302 22302 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40059
W20260619 01:12:08.573076 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22240: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:12:08.573225 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 22240
W20260619 01:12:08.590301 22225 connection.cc:535] client connection to 127.0.97.62:41275 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:41275): Connection reset by peer (error 104)
2026-06-19T01:12:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1899 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:12:08 dist-test-slave-5p8w krb5kdc[22338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:12:08 dist-test-slave-5p8w krb5kdc[22338](info): set up 2 sockets
Jun 19 01:12:08 dist-test-slave-5p8w krb5kdc[22338](info): commencing operation
krb5kdc: starting...
W20260619 01:12:09.685793 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:12:09 dist-test-slave-5p8w krb5kdc[22338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:12:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:12:09.945240 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39143
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:12:10.288149 22351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:10.288483 22351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:10.288643 22351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:10.296555 22351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:10.296746 22351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:10.296866 22351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:10.296988 22351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:10.297055 22351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:10.307428 22351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:10.308821 22351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:10.310165 22351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:10.321920 22356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:10.322068 22357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:10.322039 22359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:10.323298 22351 server_base.cc:1034] running on GCE node
I20260619 01:12:10.324128 22351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:10.325707 22351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:10.326932 22351 hybrid_clock.cc:648] HybridClock initialized: now 1781831530326891 us; error 40 us; skew 500 ppm
I20260619 01:12:10.327303 22351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:10 dist-test-slave-5p8w krb5kdc[22338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:10.334916 22351 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:12:10.336632 22351 webserver.cc:460] Webserver started at http://127.0.97.62:41355/ using document root <none> and password file <none>
I20260619 01:12:10.337481 22351 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:10.337633 22351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:10.337957 22351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:10.341167 22351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "f382896cf61a450891876feaa4d0c003"
format_stamp: "Formatted at 2026-06-19 01:12:10 on dist-test-slave-5p8w"
server_key: "6b042fa44b31a13b40e2cf779d933f82"
server_key_iv: "edd9ca05cc1fe2c5630cbe4aac2ddd41"
server_key_version: "encryptionkey@0"
I20260619 01:12:10.342067 22351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "f382896cf61a450891876feaa4d0c003"
format_stamp: "Formatted at 2026-06-19 01:12:10 on dist-test-slave-5p8w"
server_key: "6b042fa44b31a13b40e2cf779d933f82"
server_key_iv: "edd9ca05cc1fe2c5630cbe4aac2ddd41"
server_key_version: "encryptionkey@0"
I20260619 01:12:10.347784 22351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:12:10.351619 22366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:10.353147 22351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:12:10.353401 22351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "f382896cf61a450891876feaa4d0c003"
format_stamp: "Formatted at 2026-06-19 01:12:10 on dist-test-slave-5p8w"
server_key: "6b042fa44b31a13b40e2cf779d933f82"
server_key_iv: "edd9ca05cc1fe2c5630cbe4aac2ddd41"
server_key_version: "encryptionkey@0"
I20260619 01:12:10.353632 22351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:10.359575 22351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:10.363989 22351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:10.364310 22351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:10.383579 22351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39143
I20260619 01:12:10.383618 22417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39143 every 8 connection(s)
I20260619 01:12:10.385076 22351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:12:10.388829 22418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:10.390728 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 22351
I20260619 01:12:10.390905 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:12:10.391429 388 external_mini_cluster.cc:1428] Setting key 412e058e611b8b116ac8e55db7b915a8
I20260619 01:12:10.399616 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:10.393435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59981 (local address 127.0.97.62:39143)
0619 01:12:10.394315 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:10.394328 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:10.394372 (+ 44us) server_negotiation.cc:407] Connection header received
0619 01:12:10.396002 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:10.396027 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:10.396428 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:10.396866 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:10.397971 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:10.397977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:10.397991 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:10.398008 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:10.398062 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:10.398081 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:10.398334 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:10.398490 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 01:12:10.398683 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":402,"threads_started":1}
I20260619 01:12:10.403621 22418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003: Bootstrap starting.
I20260619 01:12:10.403656 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:12:10.406993 22418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:10.408207 22418 log.cc:824] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:10.411175 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:10.408238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58029 (local address 127.0.97.62:39143)
0619 01:12:10.408478 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:10.408488 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:10.409074 (+ 586us) server_negotiation.cc:407] Connection header received
0619 01:12:10.409332 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:10.409338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:10.409401 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:10.409517 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:10.410494 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:10.410499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:10.410503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:10.410519 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:10.410560 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:10.410576 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:10.410699 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:10.410840 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 01:12:10.410906 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 01:12:10.411548 22418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003: No bootstrap required, opened a new log
I20260619 01:12:10.419697 22418 raft_consensus.cc:348] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f382896cf61a450891876feaa4d0c003" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39143 } }
I20260619 01:12:10.420118 22418 raft_consensus.cc:374] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:12:10.420225 22418 raft_consensus.cc:729] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f382896cf61a450891876feaa4d0c003, State: Initialized, Role: FOLLOWER
I20260619 01:12:10.420935 22418 consensus_queue.cc:260] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f382896cf61a450891876feaa4d0c003" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39143 } }
I20260619 01:12:10.421245 22418 raft_consensus.cc:388] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:12:10.421491 22418 raft_consensus.cc:482] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:12:10.421686 22418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:10.423943 22418 raft_consensus.cc:504] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f382896cf61a450891876feaa4d0c003" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39143 } }
I20260619 01:12:10.424537 22418 leader_election.cc:302] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f382896cf61a450891876feaa4d0c003; no voters:
I20260619 01:12:10.425158 22418 leader_election.cc:288] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:12:10.425354 22428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:12:10.426070 22428 raft_consensus.cc:686] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [term 1 LEADER]: Becoming Leader. State: Replica: f382896cf61a450891876feaa4d0c003, State: Running, Role: LEADER
I20260619 01:12:10.426609 22428 consensus_queue.cc:237] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f382896cf61a450891876feaa4d0c003" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39143 } }
I20260619 01:12:10.427424 22418 sys_catalog.cc:564] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:10.429601 22429 sys_catalog.cc:455] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f382896cf61a450891876feaa4d0c003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f382896cf61a450891876feaa4d0c003" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39143 } } }
I20260619 01:12:10.430089 22429 sys_catalog.cc:458] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:10.429911 22430 sys_catalog.cc:455] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f382896cf61a450891876feaa4d0c003. Latest consensus state: current_term: 1 leader_uuid: "f382896cf61a450891876feaa4d0c003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f382896cf61a450891876feaa4d0c003" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39143 } } }
I20260619 01:12:10.430358 22430 sys_catalog.cc:458] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:10.432821 22434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:12:10.437518 22434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:12:10.447189 22434 catalog_manager.cc:1347] Generated new cluster ID: c5a6f7a6cd2645cc8d62ff5509b7244c
I20260619 01:12:10.447329 22434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:12:10.462028 22434 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:12:10.462641 22434 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:12:10.475821 22434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f382896cf61a450891876feaa4d0c003: Generated new TSK 0
I20260619 01:12:10.477059 22434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:12:10.488976 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22351
W20260619 01:12:10.490440 22413 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59981
W20260619 01:12:10.491200 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:12:10.491370 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 22351
2026-06-19T01:12:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1918 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:12:10 dist-test-slave-5p8w krb5kdc[22449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:12:10 dist-test-slave-5p8w krb5kdc[22449](info): set up 2 sockets
Jun 19 01:12:10 dist-test-slave-5p8w krb5kdc[22449](info): commencing operation
krb5kdc: starting...
W20260619 01:12:11.609814 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:12:11 dist-test-slave-5p8w krb5kdc[22449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:12:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:12:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:12:11.872615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37417
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:12:12.234869 22462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:12.235253 22462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:12.235422 22462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:12.242766 22462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:12.242920 22462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:12.243033 22462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:12.243145 22462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:12.243232 22462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:12.253711 22462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:12.255098 22462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:12.256417 22462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:12.268904 22467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:12.268972 22468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:12.269002 22470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:12.269752 22462 server_base.cc:1034] running on GCE node
I20260619 01:12:12.270530 22462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:12.271955 22462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:12.273193 22462 hybrid_clock.cc:648] HybridClock initialized: now 1781831532273139 us; error 50 us; skew 500 ppm
I20260619 01:12:12.273574 22462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:12 dist-test-slave-5p8w krb5kdc[22449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:12.276607 22462 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:12:12.278126 22462 webserver.cc:460] Webserver started at http://127.0.97.62:38391/ using document root <none> and password file <none>
I20260619 01:12:12.278851 22462 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:12.278997 22462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:12.279316 22462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:12.282342 22462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "2ad298a196c04dde82abe906f490dc72"
format_stamp: "Formatted at 2026-06-19 01:12:12 on dist-test-slave-5p8w"
server_key: "f64849a1012211413ce43b7ce81efa86"
server_key_iv: "8ec9d1c40c0dd61a487f571d093e7ab4"
server_key_version: "encryptionkey@0"
I20260619 01:12:12.283116 22462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "2ad298a196c04dde82abe906f490dc72"
format_stamp: "Formatted at 2026-06-19 01:12:12 on dist-test-slave-5p8w"
server_key: "f64849a1012211413ce43b7ce81efa86"
server_key_iv: "8ec9d1c40c0dd61a487f571d093e7ab4"
server_key_version: "encryptionkey@0"
I20260619 01:12:12.288278 22462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:12:12.291832 22477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:12.293119 22462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 01:12:12.293376 22462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "2ad298a196c04dde82abe906f490dc72"
format_stamp: "Formatted at 2026-06-19 01:12:12 on dist-test-slave-5p8w"
server_key: "f64849a1012211413ce43b7ce81efa86"
server_key_iv: "8ec9d1c40c0dd61a487f571d093e7ab4"
server_key_version: "encryptionkey@0"
I20260619 01:12:12.293589 22462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:12.311844 22462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:12.316082 22462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:12.316395 22462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:12.335510 22462 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37417
I20260619 01:12:12.335529 22528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37417 every 8 connection(s)
I20260619 01:12:12.336927 22462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:12:12.339874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 22462
I20260619 01:12:12.340054 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:12:12.340512 388 external_mini_cluster.cc:1428] Setting key dc62638b2b083b6b16ce1156c234d0ac
I20260619 01:12:12.340947 22529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:12:12 dist-test-slave-5p8w krb5kdc[22449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:12:12.355367 22529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72: Bootstrap starting.
I20260619 01:12:12.359721 22529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:12.361081 22529 log.cc:824] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:12.364210 22529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72: No bootstrap required, opened a new log
I20260619 01:12:12.364590 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:12.342316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34445 (local address 127.0.97.62:37417)
0619 01:12:12.343104 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:12.343119 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:12.343166 (+ 47us) server_negotiation.cc:407] Connection header received
0619 01:12:12.344827 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:12.344854 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:12.345464 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:12.345908 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:12.347202 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:12.349325 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:12.350760 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:12.351785 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:12.351970 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:12.356072 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:12.356083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:12.356103 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:12.356158 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:12.360228 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:12.360972 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:12.360981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:12.360989 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:12.361238 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:12.361671 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:12.361677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:12.361680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:12.362298 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:12.362624 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:12.363085 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 01:12:12.363546 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":338,"threads_started":1}
I20260619 01:12:12.368669 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:12:12.372735 22529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad298a196c04dde82abe906f490dc72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } }
I20260619 01:12:12.373438 22529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:12:12.373554 22529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad298a196c04dde82abe906f490dc72, State: Initialized, Role: FOLLOWER
I20260619 01:12:12.374236 22529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad298a196c04dde82abe906f490dc72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } }
I20260619 01:12:12.374472 22529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:12:12.374626 22529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:12:12.374814 22529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:12.377103 22529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad298a196c04dde82abe906f490dc72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } }
I20260619 01:12:12.377663 22529 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ad298a196c04dde82abe906f490dc72; no voters:
I20260619 01:12:12.378398 22529 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:12:12.378549 22539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:12:12.380162 22539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [term 1 LEADER]: Becoming Leader. State: Replica: 2ad298a196c04dde82abe906f490dc72, State: Running, Role: LEADER
I20260619 01:12:12.380831 22539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad298a196c04dde82abe906f490dc72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } }
I20260619 01:12:12.380998 22529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:12.388309 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:12.373038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33145 (local address 127.0.97.62:37417)
0619 01:12:12.373246 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:12.373255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:12.373726 (+ 471us) server_negotiation.cc:407] Connection header received
0619 01:12:12.373969 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:12.373979 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:12.374045 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:12.374151 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:12.375738 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:12.376501 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:12.377988 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:12.378584 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:12.378783 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:12.381826 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:12.381837 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:12.381841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:12.381874 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:12.385088 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:12.385889 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:12.385895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:12.385898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:12.386043 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:12.386444 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:12.386449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:12.386452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:12.386749 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:12.387041 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:12.387872 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 01:12:12.388035 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 01:12:12.391530 22540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ad298a196c04dde82abe906f490dc72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad298a196c04dde82abe906f490dc72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } } }
I20260619 01:12:12.391964 22540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:12.392488 22541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ad298a196c04dde82abe906f490dc72. Latest consensus state: current_term: 1 leader_uuid: "2ad298a196c04dde82abe906f490dc72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad298a196c04dde82abe906f490dc72" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37417 } } }
I20260619 01:12:12.392825 22541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:12.394234 22549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:12:12.401001 22549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:12:12.410172 22549 catalog_manager.cc:1347] Generated new cluster ID: 03173dabf10642978259ad5dd3c1b59f
I20260619 01:12:12.410295 22549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:12:12.428557 22549 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:12:12.430290 22549 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:12:12.438412 22549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ad298a196c04dde82abe906f490dc72: Generated new TSK 0
I20260619 01:12:12.439515 22549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:12:12.457971 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22462
I20260619 01:12:12.459277 22524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:12:12.552301 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 22462
2026-06-19T01:12:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2061 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:12:12 dist-test-slave-5p8w krb5kdc[22561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:12:12 dist-test-slave-5p8w krb5kdc[22561](info): set up 2 sockets
Jun 19 01:12:12 dist-test-slave-5p8w krb5kdc[22561](info): commencing operation
krb5kdc: starting...
W20260619 01:12:13.653880 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:12:13 dist-test-slave-5p8w krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:12:13.908013 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46099
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:12:14.244004 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:14.244308 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:14.244488 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:14.251921 22574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:14.252058 22574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:14.252197 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:14.252275 22574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:14.252368 22574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:14.262662 22574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:14.264034 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:14.265413 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:14.278056 22582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:14.278154 22579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:14.278155 22580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:14.279237 22574 server_base.cc:1034] running on GCE node
I20260619 01:12:14.280056 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:14.281553 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:14.282788 22574 hybrid_clock.cc:648] HybridClock initialized: now 1781831534282747 us; error 58 us; skew 500 ppm
I20260619 01:12:14.283149 22574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:14 dist-test-slave-5p8w krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:14.286317 22574 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:12:14.287866 22574 webserver.cc:460] Webserver started at http://127.0.97.62:35889/ using document root <none> and password file <none>
I20260619 01:12:14.288558 22574 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:14.288712 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:14.289016 22574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:14.292186 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "ac4a9c6edee9478a87a2a6d044eb59d2"
format_stamp: "Formatted at 2026-06-19 01:12:14 on dist-test-slave-5p8w"
server_key: "59cef11a4efb38cc789eac91a3a6b865"
server_key_iv: "48ca94b36489030a7af8954e65900189"
server_key_version: "encryptionkey@0"
I20260619 01:12:14.293010 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "ac4a9c6edee9478a87a2a6d044eb59d2"
format_stamp: "Formatted at 2026-06-19 01:12:14 on dist-test-slave-5p8w"
server_key: "59cef11a4efb38cc789eac91a3a6b865"
server_key_iv: "48ca94b36489030a7af8954e65900189"
server_key_version: "encryptionkey@0"
I20260619 01:12:14.298323 22574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 01:12:14.301702 22589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:14.303020 22574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:12:14.303198 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "ac4a9c6edee9478a87a2a6d044eb59d2"
format_stamp: "Formatted at 2026-06-19 01:12:14 on dist-test-slave-5p8w"
server_key: "59cef11a4efb38cc789eac91a3a6b865"
server_key_iv: "48ca94b36489030a7af8954e65900189"
server_key_version: "encryptionkey@0"
I20260619 01:12:14.303392 22574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:14.332844 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:14.337486 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:14.337795 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:14.355247 22574 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46099
I20260619 01:12:14.355288 22640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46099 every 8 connection(s)
I20260619 01:12:14.356595 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:12:14.360339 22641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:14.364527 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 22574
I20260619 01:12:14.364720 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:12:14.365124 388 external_mini_cluster.cc:1428] Setting key 73e4db3064d112e652b486bb898c924f
I20260619 01:12:14.372404 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2: Bootstrap starting.
I20260619 01:12:14.376152 22641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:14.377501 22641 log.cc:824] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:14.380755 22641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2: No bootstrap required, opened a new log
I20260619 01:12:14.381232 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:14.367201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44961 (local address 127.0.97.62:46099)
0619 01:12:14.368241 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:14.368260 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:14.368675 (+ 415us) server_negotiation.cc:407] Connection header received
0619 01:12:14.371210 (+ 2535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:14.371238 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:14.371768 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:14.372258 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:14.372770 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:14.374828 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:14.376183 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:14.377261 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:14.377470 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:14.378598 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:14.378606 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:14.378619 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:14.379421 (+ 802us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:14.379504 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:14.379529 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:14.379681 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:14.379926 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 01:12:14.380190 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":446,"threads_started":1}
I20260619 01:12:14.385259 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:12:14.389494 22641 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4a9c6edee9478a87a2a6d044eb59d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46099 } }
I20260619 01:12:14.389986 22641 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:12:14.390175 22641 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4a9c6edee9478a87a2a6d044eb59d2, State: Initialized, Role: FOLLOWER
I20260619 01:12:14.390919 22641 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4a9c6edee9478a87a2a6d044eb59d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46099 } }
I20260619 01:12:14.391230 22641 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:12:14.391379 22641 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:12:14.391577 22641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:14.394708 22641 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4a9c6edee9478a87a2a6d044eb59d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46099 } }
I20260619 01:12:14.395311 22641 leader_election.cc:302] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac4a9c6edee9478a87a2a6d044eb59d2; no voters:
I20260619 01:12:14.396050 22641 leader_election.cc:288] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:12:14.396189 22651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:12:14.397056 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:14.389932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43363 (local address 127.0.97.62:46099)
0619 01:12:14.390151 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:14.390165 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:14.390619 (+ 454us) server_negotiation.cc:407] Connection header received
0619 01:12:14.390850 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:14.390853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:14.390919 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:14.390998 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:14.391702 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:14.392452 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:14.394328 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:14.394779 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:14.394916 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:14.396125 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:14.396132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:14.396138 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:14.396177 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:14.396226 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:14.396250 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:14.396355 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:14.396616 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 01:12:14.396722 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:12:14.397693 22651 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [term 1 LEADER]: Becoming Leader. State: Replica: ac4a9c6edee9478a87a2a6d044eb59d2, State: Running, Role: LEADER
I20260619 01:12:14.398308 22651 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4a9c6edee9478a87a2a6d044eb59d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46099 } }
I20260619 01:12:14.399181 22641 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:14.401434 22652 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac4a9c6edee9478a87a2a6d044eb59d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4a9c6edee9478a87a2a6d044eb59d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46099 } } }
I20260619 01:12:14.401780 22653 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4a9c6edee9478a87a2a6d044eb59d2. Latest consensus state: current_term: 1 leader_uuid: "ac4a9c6edee9478a87a2a6d044eb59d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4a9c6edee9478a87a2a6d044eb59d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46099 } } }
I20260619 01:12:14.402410 22652 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:14.402695 22653 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:14.403698 22660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:12:14.410171 22660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:12:14.423110 22660 catalog_manager.cc:1347] Generated new cluster ID: c03764d45d84459daf616eda5e1a1127
I20260619 01:12:14.423240 22660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:12:14.448130 22660 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:12:14.449719 22660 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:12:14.463879 22660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac4a9c6edee9478a87a2a6d044eb59d2: Generated new TSK 0
I20260619 01:12:14.464968 22660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:12:14.480003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22574
W20260619 01:12:14.481070 22636 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44961
W20260619 01:12:14.481866 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22574: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:12:14.482019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 22574
2026-06-19T01:12:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:12:14 dist-test-slave-5p8w krb5kdc[22672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:12:14 dist-test-slave-5p8w krb5kdc[22672](info): set up 2 sockets
Jun 19 01:12:14 dist-test-slave-5p8w krb5kdc[22672](info): commencing operation
krb5kdc: starting...
W20260619 01:12:15.574186 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:12:15 dist-test-slave-5p8w krb5kdc[22672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:12:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:12:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:12:15.838766 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35367
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:12:16.172787 22685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:16.173096 22685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:16.173190 22685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:16.180650 22685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:16.180787 22685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:16.180835 22685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:16.180888 22685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:16.180928 22685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:16.191181 22685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:16.192633 22685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:16.194351 22685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:16.207142 22690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:16.207111 22691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:16.207072 22693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:16.207999 22685 server_base.cc:1034] running on GCE node
I20260619 01:12:16.208853 22685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:16.210477 22685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:16.211694 22685 hybrid_clock.cc:648] HybridClock initialized: now 1781831536211618 us; error 60 us; skew 500 ppm
I20260619 01:12:16.212072 22685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:16 dist-test-slave-5p8w krb5kdc[22672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:16.218611 22685 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:12:16.220278 22685 webserver.cc:460] Webserver started at http://127.0.97.62:43767/ using document root <none> and password file <none>
I20260619 01:12:16.221149 22685 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:16.221271 22685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:16.221614 22685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:16.224772 22685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "2dfc12119ea0421498d5c1880baa6a32"
format_stamp: "Formatted at 2026-06-19 01:12:16 on dist-test-slave-5p8w"
server_key: "b50490633896183afe9402674aa83824"
server_key_iv: "19608432208014a72ad44eab395f378a"
server_key_version: "encryptionkey@0"
I20260619 01:12:16.225665 22685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "2dfc12119ea0421498d5c1880baa6a32"
format_stamp: "Formatted at 2026-06-19 01:12:16 on dist-test-slave-5p8w"
server_key: "b50490633896183afe9402674aa83824"
server_key_iv: "19608432208014a72ad44eab395f378a"
server_key_version: "encryptionkey@0"
I20260619 01:12:16.231125 22685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:12:16.234826 22700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:16.236222 22685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:12:16.236426 22685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "2dfc12119ea0421498d5c1880baa6a32"
format_stamp: "Formatted at 2026-06-19 01:12:16 on dist-test-slave-5p8w"
server_key: "b50490633896183afe9402674aa83824"
server_key_iv: "19608432208014a72ad44eab395f378a"
server_key_version: "encryptionkey@0"
I20260619 01:12:16.236639 22685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:16.242599 22685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:16.247160 22685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:16.247480 22685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:16.266467 22685 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35367
I20260619 01:12:16.266505 22751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35367 every 8 connection(s)
I20260619 01:12:16.267966 22685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:12:16.271739 22752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:16.274296 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 22685
I20260619 01:12:16.274492 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:12:16.274930 388 external_mini_cluster.cc:1428] Setting key 9f2eba4912bc3210d4be284d6082120e
I20260619 01:12:16.283022 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:16.276735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43289 (local address 127.0.97.62:35367)
0619 01:12:16.277688 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:16.277700 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:16.277757 (+ 57us) server_negotiation.cc:407] Connection header received
0619 01:12:16.279423 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:16.279445 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:16.279805 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:16.280190 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:16.281340 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:16.281350 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:16.281370 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:16.281392 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:16.281456 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:16.281481 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:16.281732 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:16.281894 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 01:12:16.282075 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":342,"threads_started":1}
I20260619 01:12:16.284384 22752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32: Bootstrap starting.
I20260619 01:12:16.286734 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:12:16.287726 22752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:16.288769 22752 log.cc:824] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:16.291988 22752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32: No bootstrap required, opened a new log
I20260619 01:12:16.294061 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:16.291375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42241 (local address 127.0.97.62:35367)
0619 01:12:16.291604 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:16.291610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:16.291974 (+ 364us) server_negotiation.cc:407] Connection header received
0619 01:12:16.292218 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:16.292224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:16.292307 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:16.292417 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:16.293398 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:16.293405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:16.293416 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:12:16.293436 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:16.293485 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:12:16.293504 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:16.293625 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:16.293705 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 01:12:16.293769 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 01:12:16.300782 22752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc12119ea0421498d5c1880baa6a32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35367 } }
I20260619 01:12:16.301282 22752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:12:16.301525 22752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dfc12119ea0421498d5c1880baa6a32, State: Initialized, Role: FOLLOWER
I20260619 01:12:16.302233 22752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc12119ea0421498d5c1880baa6a32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35367 } }
I20260619 01:12:16.302527 22752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:12:16.302688 22752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:12:16.302872 22752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:16.305212 22752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc12119ea0421498d5c1880baa6a32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35367 } }
I20260619 01:12:16.305805 22752 leader_election.cc:302] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dfc12119ea0421498d5c1880baa6a32; no voters:
I20260619 01:12:16.306541 22752 leader_election.cc:288] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:12:16.306663 22762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:12:16.307390 22762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [term 1 LEADER]: Becoming Leader. State: Replica: 2dfc12119ea0421498d5c1880baa6a32, State: Running, Role: LEADER
I20260619 01:12:16.308022 22762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc12119ea0421498d5c1880baa6a32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35367 } }
I20260619 01:12:16.309013 22752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:16.311218 22764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dfc12119ea0421498d5c1880baa6a32. Latest consensus state: current_term: 1 leader_uuid: "2dfc12119ea0421498d5c1880baa6a32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc12119ea0421498d5c1880baa6a32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35367 } } }
I20260619 01:12:16.311705 22764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:16.311232 22763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dfc12119ea0421498d5c1880baa6a32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfc12119ea0421498d5c1880baa6a32" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35367 } } }
I20260619 01:12:16.312175 22763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32 [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:16.313421 22769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:12:16.319253 22769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:12:16.328030 22769 catalog_manager.cc:1347] Generated new cluster ID: dae2b9425e59483e8a63dd638b77ebd1
I20260619 01:12:16.328155 22769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:12:16.360409 22769 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:12:16.360878 22769 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:12:16.366430 22769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dfc12119ea0421498d5c1880baa6a32: Generated new TSK 0
I20260619 01:12:16.367318 22769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:12:16.385514 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22685
W20260619 01:12:16.386883 22747 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43289
W20260619 01:12:16.387669 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22685: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:12:16.387833 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 22685
2026-06-19T01:12:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1905 ms)
[----------] 18 tests from AuthTokenIssuingTest (35260 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:12:16 dist-test-slave-5p8w krb5kdc[22783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:12:16 dist-test-slave-5p8w krb5kdc[22783](info): set up 2 sockets
Jun 19 01:12:16 dist-test-slave-5p8w krb5kdc[22783](info): commencing operation
krb5kdc: starting...
W20260619 01:12:17.489975 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:12:17 dist-test-slave-5p8w krb5kdc[22783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:12:17.754398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44293
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44293,127.0.97.61:43239,127.0.97.60:45965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:12:18.081789 22796 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:12:18.082194 22796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:18.082316 22796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:18.082494 22796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:18.089768 22796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:18.089857 22796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:18.089982 22796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:18.090070 22796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:18.090160 22796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:18.099908 22796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44293,127.0.97.61:43239,127.0.97.60:45965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:18.101226 22796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:18.102550 22796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:18.114171 22801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:18.114128 22802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:18.114205 22804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:18.114893 22796 server_base.cc:1034] running on GCE node
I20260619 01:12:18.115578 22796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:18.116881 22796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:18.118075 22796 hybrid_clock.cc:648] HybridClock initialized: now 1781831538118019 us; error 26 us; skew 500 ppm
I20260619 01:12:18.118357 22796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:18 dist-test-slave-5p8w krb5kdc[22783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:18.121428 22796 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:12:18.122725 22796 webserver.cc:460] Webserver started at http://127.0.97.62:34111/ using document root <none> and password file <none>
I20260619 01:12:18.123389 22796 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:18.123471 22796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:18.123735 22796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:18.126626 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "f1108f07f1044fd080da2f8bc73301cd"
format_stamp: "Formatted at 2026-06-19 01:12:18 on dist-test-slave-5p8w"
server_key: "6bcc3ff3809737d8af5529d30b21607a"
server_key_iv: "7f41c07e7b254b469a985d2fdbcb454f"
server_key_version: "encryptionkey@0"
I20260619 01:12:18.127357 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "f1108f07f1044fd080da2f8bc73301cd"
format_stamp: "Formatted at 2026-06-19 01:12:18 on dist-test-slave-5p8w"
server_key: "6bcc3ff3809737d8af5529d30b21607a"
server_key_iv: "7f41c07e7b254b469a985d2fdbcb454f"
server_key_version: "encryptionkey@0"
I20260619 01:12:18.132304 22796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:12:18.135497 22811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:18.136768 22796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:12:18.136904 22796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "f1108f07f1044fd080da2f8bc73301cd"
format_stamp: "Formatted at 2026-06-19 01:12:18 on dist-test-slave-5p8w"
server_key: "6bcc3ff3809737d8af5529d30b21607a"
server_key_iv: "7f41c07e7b254b469a985d2fdbcb454f"
server_key_version: "encryptionkey@0"
I20260619 01:12:18.137072 22796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:18.165135 22796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:18.169118 22796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:18.169378 22796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:18.187778 22796 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44293
I20260619 01:12:18.187791 22862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44293 every 8 connection(s)
I20260619 01:12:18.189137 22796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:12:18.189549 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 22796
I20260619 01:12:18.189760 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:12:18.190572 388 external_mini_cluster.cc:1428] Setting key 41e615d9aabd1df2857f03f9210b4a50
I20260619 01:12:18.193140 22863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:18.199672 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 19 01:12:18 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:12:18.224982 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.202944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44293 (local address 127.0.0.1:35434)
0619 01:12:18.204255 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.204300 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:18.206098 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:18.207025 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:18.207030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:18.207585 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:18.207916 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:18.207936 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.210364 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.210368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:18.211425 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:18.211437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.212746 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.212757 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:18.212974 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:18.213675 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:18.213696 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:18.217223 (+ 3527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:18.220474 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:18.220486 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:18.220501 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:18.221014 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:18.221547 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:18.221554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:18.221559 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:18.221735 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:18.222743 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:18.222753 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:18.223006 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 01:12:18.223281 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 01:12:18.223816 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":74,"thread_start_us":836,"threads_started":1}
I20260619 01:12:18.225035 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.202944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.0.97.62:44293)
0619 01:12:18.203837 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:18.203854 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:18.204449 (+ 595us) server_negotiation.cc:407] Connection header received
0619 01:12:18.206463 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:18.206473 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:18.206783 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:18.206999 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:18.208132 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.210192 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.211571 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.212528 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.212740 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:18.217512 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:18.217520 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:18.217533 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:18.217564 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:18.220291 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:18.221164 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:18.221176 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:18.221183 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:18.221424 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:18.221854 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:18.221862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:18.221868 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:18.222600 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:18.222826 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:18.223443 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 01:12:18.224002 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":244,"server-negotiator.queue_time_us":644,"thread_start_us":403,"threads_started":1}
I20260619 01:12:18.227968 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:12:18.229038 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.228565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:38540)
0619 01:12:18.228757 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.228888 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 01:12:18.229336 22814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
I20260619 01:12:18.230984 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.230536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:38552)
0619 01:12:18.230717 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.230820 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 01:12:18.231178 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43239: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 01:12:18.246495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43239
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44293,127.0.97.61:43239,127.0.97.60:45965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:12:18.280555 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } attempt: 1
I20260619 01:12:18.281858 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.281361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:38564)
0619 01:12:18.281571 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.281694 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 01:12:18.282888 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.282538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:38580)
0619 01:12:18.282660 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.282743 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 01:12:18.283074 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43239: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
I20260619 01:12:18.351380 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } attempt: 2
I20260619 01:12:18.352830 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.352124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:38586)
0619 01:12:18.352451 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.352591 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 01:12:18.354035 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.353560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:38592)
0619 01:12:18.353742 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.353838 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 01:12:18.354259 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43239: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
I20260619 01:12:18.445554 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } attempt: 3
I20260619 01:12:18.446738 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.446227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:39760)
0619 01:12:18.446460 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.446570 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 01:12:18.447799 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.447404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:39774)
0619 01:12:18.447555 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.447637 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 01:12:18.447988 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43239: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
W20260619 01:12:18.582471 22870 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:12:18.582912 22870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:18.583024 22870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:18.583197 22870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:18.590801 22870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:18.590907 22870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:18.590965 22870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:18.591010 22870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:18.591075 22870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:18.591240 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } attempt: 4
I20260619 01:12:18.592554 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.591928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:39782)
0619 01:12:18.592288 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.592395 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260619 01:12:18.593793 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.593390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:39790)
0619 01:12:18.593545 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.593639 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 01:12:18.593990 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43239: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43239: connect: Connection refused (error 111)
I20260619 01:12:18.602214 22870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44293,127.0.97.61:43239,127.0.97.60:45965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:18.603607 22870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:18.605007 22870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:18.617678 22878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:18.617722 22876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:18.617657 22875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:18.623458 22870 server_base.cc:1034] running on GCE node
I20260619 01:12:18.624429 22870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:18.626176 22870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:18.627429 22870 hybrid_clock.cc:648] HybridClock initialized: now 1781831538627354 us; error 37 us; skew 500 ppm
I20260619 01:12:18.627859 22870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:18 dist-test-slave-5p8w krb5kdc[22783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:18.631358 22870 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 01:12:18.633074 22870 webserver.cc:460] Webserver started at http://127.0.97.61:45603/ using document root <none> and password file <none>
I20260619 01:12:18.633932 22870 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:18.634023 22870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:18.634394 22870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:18.637603 22870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/data/instance:
uuid: "c465347475514503a9c93ab9d303edf3"
format_stamp: "Formatted at 2026-06-19 01:12:18 on dist-test-slave-5p8w"
server_key: "e12d8e4caa9c87287657734c108cba5c"
server_key_iv: "9d448bf250364b0604320d04d6518d3f"
server_key_version: "encryptionkey@0"
I20260619 01:12:18.638406 22870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/wal/instance:
uuid: "c465347475514503a9c93ab9d303edf3"
format_stamp: "Formatted at 2026-06-19 01:12:18 on dist-test-slave-5p8w"
server_key: "e12d8e4caa9c87287657734c108cba5c"
server_key_iv: "9d448bf250364b0604320d04d6518d3f"
server_key_version: "encryptionkey@0"
I20260619 01:12:18.643806 22870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:12:18.647338 22885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:18.648741 22870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:12:18.648929 22870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/wal
uuid: "c465347475514503a9c93ab9d303edf3"
format_stamp: "Formatted at 2026-06-19 01:12:18 on dist-test-slave-5p8w"
server_key: "e12d8e4caa9c87287657734c108cba5c"
server_key_iv: "9d448bf250364b0604320d04d6518d3f"
server_key_version: "encryptionkey@0"
I20260619 01:12:18.649158 22870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:18.681047 22870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:18.685722 22870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:18.686012 22870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:18.705348 22870 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43239
I20260619 01:12:18.705397 22936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43239 every 8 connection(s)
I20260619 01:12:18.706797 22870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
I20260619 01:12:18.711002 22937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:18.712021 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 22870
I20260619 01:12:18.712213 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-1/wal/instance
I20260619 01:12:18.712787 388 external_mini_cluster.cc:1428] Setting key cb07a46680b6ad025c7d59663aa69076
I20260619 01:12:18.721774 22937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:12:18 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 01:12:18.744081 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.725217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.0.97.62:44293)
0619 01:12:18.725425 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:18.725430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:18.726881 (+ 1451us) server_negotiation.cc:407] Connection header received
0619 01:12:18.728894 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:18.728900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:18.728998 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:18.729115 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:18.730785 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.731516 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.733122 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.733601 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.733737 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:18.737425 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:18.737431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:18.737434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:18.737471 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:18.740316 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:18.741271 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:18.741276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:18.741278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:18.741496 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:18.741966 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:18.741974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:18.741980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:18.742426 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:18.742594 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:18.743371 (+ 777us) server_negotiation.cc:299] Negotiation successful
0619 01:12:18.743651 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:12:18.744807 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.725888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44293 (local address 127.0.0.1:40286)
0619 01:12:18.726702 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.726741 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:18.728571 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:18.729441 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:18.729450 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:18.730008 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:18.730597 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:18.730614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.731692 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.731696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:18.732960 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:18.732973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.733861 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.733869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:18.734111 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:18.734881 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:18.734913 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:18.737137 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:18.740554 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:18.740562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:18.740578 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:18.741133 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:18.741633 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:18.741637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:18.741639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:18.741843 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:18.742580 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:18.742586 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:18.742980 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 01:12:18.743226 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 01:12:18.743643 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":356,"threads_started":1}
I20260619 01:12:18.746660 22937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:12:18 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:12:18.764248 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.747351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:39798)
0619 01:12:18.748117 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.748142 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:18.748251 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:18.748714 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:18.748722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:18.749277 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:18.749522 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:18.749546 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.751474 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.751478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:18.752346 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:18.752356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.753701 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.753706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:18.753874 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:18.754507 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:18.754524 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:18.756839 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:18.761451 (+ 4612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:18.761455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:18.761466 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:18.761869 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:18.762352 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:18.762356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:18.762359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:18.762557 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:18.763384 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:18.763388 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:18.763543 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 01:12:18.763687 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 01:12:18.763919 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614}
I20260619 01:12:18.764910 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.747620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.0.97.61:43239)
0619 01:12:18.748303 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:18.748309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:18.748334 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:12:18.748379 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:18.748384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:18.748563 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:18.748674 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:18.749707 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.751298 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.752494 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.753515 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.753658 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:18.757082 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:18.757100 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:18.757121 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:18.757165 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:18.761269 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:18.762015 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:18.762027 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:18.762033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:18.762194 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:18.762683 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:18.762691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:18.762697 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:18.763229 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:18.763452 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:18.763880 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 01:12:18.764183 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":419,"threads_started":1}
I20260619 01:12:18.767855 22937 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:12:18.769129 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.768641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:36908)
0619 01:12:18.768821 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.768974 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 01:12:18.769475 22886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
I20260619 01:12:18.771135 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.770731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:36922)
0619 01:12:18.770901 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.771010 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 01:12:18.771348 22937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45965: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 01:12:18.776737 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45965
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44293,127.0.97.61:43239,127.0.97.60:45965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:12:18.820680 22937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } attempt: 1
I20260619 01:12:18.822009 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.821495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:36938)
0619 01:12:18.821742 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.821837 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 01:12:18.823150 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.822775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:36946)
0619 01:12:18.822953 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.823031 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 01:12:18.823372 22937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45965: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
I20260619 01:12:18.891611 22937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } attempt: 2
I20260619 01:12:18.892778 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.892314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:36962)
0619 01:12:18.892522 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.892623 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 01:12:18.893250 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } attempt: 5
I20260619 01:12:18.893913 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.893502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:36978)
0619 01:12:18.893650 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.893726 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 01:12:18.894246 22937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45965: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Jun 19 01:12:18 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:12:18.907341 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.894089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.0.97.61:43239)
0619 01:12:18.894351 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:18.894359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:18.894379 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:12:18.894444 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:18.894447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:18.894515 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:18.894603 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:18.895594 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.896382 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.898053 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.898480 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.898604 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:18.902067 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:18.902072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:18.902075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:18.902107 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:18.904718 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:18.905420 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:18.905427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:18.905430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:18.905568 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:18.905956 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:18.905960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:18.905962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:18.906267 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:18.906419 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:18.906901 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 01:12:18.907090 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260619 01:12:18.907384 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.893869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:39808)
0619 01:12:18.894097 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.894115 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:18.894232 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:18.894683 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:18.894687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:18.895176 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:18.895408 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:18.895419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.896534 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.896540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:18.897886 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:18.897910 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:18.898649 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:18.898654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:18.898869 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:18.899482 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:18.899505 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:18.901876 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:18.904859 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:18.904865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:18.904870 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:18.905285 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:18.905685 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:18.905690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:18.905694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:18.905835 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:18.906411 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:18.906417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:18.906585 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 01:12:18.906752 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 01:12:18.907037 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 01:12:18.908473 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:12:18.909893 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.909272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:36992)
0619 01:12:18.909617 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.909716 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260619 01:12:18.910890 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.910532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37002)
0619 01:12:18.910710 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.910788 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 01:12:18.911123 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45965: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
I20260619 01:12:18.962402 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } attempt: 1
I20260619 01:12:18.963698 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.963172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37014)
0619 01:12:18.963393 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.963512 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 01:12:18.964890 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.964546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37024)
0619 01:12:18.964702 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.964778 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 01:12:18.965160 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45965: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
I20260619 01:12:18.985517 22937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } attempt: 3
I20260619 01:12:18.986676 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.986204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37032)
0619 01:12:18.986406 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.986506 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 01:12:18.987751 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:18.987330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37046)
0619 01:12:18.987558 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:18.987631 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260619 01:12:18.987954 22937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45965: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
I20260619 01:12:19.033435 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } attempt: 2
I20260619 01:12:19.034664 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.034212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37054)
0619 01:12:19.034410 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:19.034499 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 01:12:19.035676 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.035320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37070)
0619 01:12:19.035501 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:19.035570 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 01:12:19.035921 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45965: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
W20260619 01:12:19.116030 22944 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:12:19.116382 22944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:19.116447 22944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:19.116603 22944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:19.124135 22944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:19.124233 22944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:19.124327 22944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:19.124430 22944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:19.124537 22944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:19.131147 22937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } attempt: 4
I20260619 01:12:19.132711 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.131983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37084)
0619 01:12:19.132315 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:19.132446 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 01:12:19.134016 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.133530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37096)
0619 01:12:19.133800 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:19.133886 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260619 01:12:19.134307 22937 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45965: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
I20260619 01:12:19.138378 22944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44293,127.0.97.61:43239,127.0.97.60:45965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:19.139724 22944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:19.141247 22944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 01:12:19.142143 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } attempt: 3
I20260619 01:12:19.143172 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.142736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37110)
0619 01:12:19.142938 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:19.143022 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 01:12:19.144187 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.143814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37122)
0619 01:12:19.144004 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:19.144074 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260619 01:12:19.144411 22863 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45965: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45965: connect: Connection refused (error 111)
W20260619 01:12:19.154024 22950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:19.154089 22949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:19.154122 22952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:19.155099 22944 server_base.cc:1034] running on GCE node
I20260619 01:12:19.155897 22944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:19.157402 22944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:19.158660 22944 hybrid_clock.cc:648] HybridClock initialized: now 1781831539158597 us; error 55 us; skew 500 ppm
I20260619 01:12:19.159013 22944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:19 dist-test-slave-5p8w krb5kdc[22783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:19.162310 22944 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 01:12:19.163888 22944 webserver.cc:460] Webserver started at http://127.0.97.60:44017/ using document root <none> and password file <none>
I20260619 01:12:19.164644 22944 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:19.164784 22944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:19.165114 22944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:19.168284 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/data/instance:
uuid: "58c69295cec64b58835800db72fe3ed1"
format_stamp: "Formatted at 2026-06-19 01:12:19 on dist-test-slave-5p8w"
server_key: "26a93c0c22c389a0206079f948c4fff1"
server_key_iv: "e76eccb81e7b996751f9fcb34690e983"
server_key_version: "encryptionkey@0"
I20260619 01:12:19.169112 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/wal/instance:
uuid: "58c69295cec64b58835800db72fe3ed1"
format_stamp: "Formatted at 2026-06-19 01:12:19 on dist-test-slave-5p8w"
server_key: "26a93c0c22c389a0206079f948c4fff1"
server_key_iv: "e76eccb81e7b996751f9fcb34690e983"
server_key_version: "encryptionkey@0"
I20260619 01:12:19.174446 22944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 01:12:19.177976 22959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:19.179370 22944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:12:19.179562 22944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/wal
uuid: "58c69295cec64b58835800db72fe3ed1"
format_stamp: "Formatted at 2026-06-19 01:12:19 on dist-test-slave-5p8w"
server_key: "26a93c0c22c389a0206079f948c4fff1"
server_key_iv: "e76eccb81e7b996751f9fcb34690e983"
server_key_version: "encryptionkey@0"
I20260619 01:12:19.179791 22944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:19.197970 22944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:19.201917 22944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:19.202193 22944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:19.220999 22944 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45965
I20260619 01:12:19.221021 23010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45965 every 8 connection(s)
I20260619 01:12:19.222570 22944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
I20260619 01:12:19.226325 23011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:19.232936 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 22944
I20260619 01:12:19.233122 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781831171286417-388-0/minicluster-data/master-2/wal/instance
I20260619 01:12:19.233595 388 external_mini_cluster.cc:1428] Setting key 0c83162608e9a38a0a4a53d362eed5db
I20260619 01:12:19.232877 23011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:12:19 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 19 01:12:19 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:12:19.253849 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.236174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.0.97.62:44293)
0619 01:12:19.236533 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.236574 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.237279 (+ 705us) server_negotiation.cc:407] Connection header received
0619 01:12:19.239083 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.239090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.239171 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.239303 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.240896 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.241718 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.243346 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.243863 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.243958 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.247841 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:19.247845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:19.247849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:19.247881 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:19.250485 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.251288 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.251295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.251299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.251439 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.251876 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.251882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.251885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.252159 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:19.252336 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.253155 (+ 819us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.253422 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260619 01:12:19.254882 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.235691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.0.97.60:45965)
0619 01:12:19.237176 (+ 1485us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.237184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.237229 (+ 45us) server_negotiation.cc:407] Connection header received
0619 01:12:19.238826 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.238846 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.239080 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.239242 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.240688 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.242235 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.243567 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.244189 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.244349 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.248421 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:19.248435 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:19.248454 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:19.248496 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:19.251422 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.252137 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.252144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.252146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.252282 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.252676 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.252685 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.252687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.253076 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:19.253418 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.253737 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.253995 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":1276,"thread_start_us":1050,"threads_started":1}
I20260619 01:12:19.254917 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.236113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44293 (local address 127.0.0.1:40288)
0619 01:12:19.237122 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:19.237167 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:19.238842 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:19.239408 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:19.239414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:19.240140 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:19.240670 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:19.240686 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.241878 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.241884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:19.243200 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:19.243211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.244081 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.244092 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:19.244312 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.245236 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:19.245258 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:19.247651 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:19.250623 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:19.250640 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:19.250653 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:19.251141 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:19.251555 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:19.251560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:19.251564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:19.251759 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:19.252315 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:19.252323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:19.252725 (+ 402us) client_negotiation.cc:769] Sending connection context
0619 01:12:19.253067 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 01:12:19.253589 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":43,"thread_start_us":538,"threads_started":1}
I20260619 01:12:19.256582 23011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:12:19 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 01:12:19 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:12:19.271519 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.257268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43239 (local address 127.0.0.1:39816)
0619 01:12:19.257449 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:19.257472 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:19.257574 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:19.257917 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:19.257920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:19.258366 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:19.258575 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:19.258586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.259982 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.259985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:19.261344 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:19.261359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.262304 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.262308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:19.262463 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.263074 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:19.263091 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:19.265472 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:19.268963 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:19.268969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:19.268973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:19.269441 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:19.269864 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:19.269869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:19.269873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:19.270015 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:19.270539 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:19.270544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:19.270695 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 01:12:19.270837 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 01:12:19.271147 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 01:12:19.271546 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.257397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.0.97.61:43239)
0619 01:12:19.257585 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.257591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.257612 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:12:19.257677 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.257682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.257773 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.257876 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.258730 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.259824 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.261496 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.262156 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.262288 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.265700 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:19.265707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:19.265714 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:19.265766 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:19.268807 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.269574 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.269580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.269584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.269744 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.270130 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.270135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.270138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.270414 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:19.270579 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.270951 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.271169 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:12:19.272469 23011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:12:19.274343 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.258461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.0.97.62:44293)
0619 01:12:19.258652 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.258660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.258678 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:12:19.258743 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.258748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.258812 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.258880 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.260300 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.261423 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.262919 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.263361 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.263500 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.268437 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:19.268443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:19.268446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:19.268483 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:19.271197 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.272058 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.272063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.272067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.272215 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.272610 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.272615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.272619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.272988 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:19.273278 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.273799 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.274043 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 01:12:19 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 01:12:19 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:12:19.287619 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.273236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.0.97.60:45965)
0619 01:12:19.273455 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.273461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.273667 (+ 206us) server_negotiation.cc:407] Connection header received
0619 01:12:19.273823 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.273828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.273895 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.273998 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.274901 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.275695 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.277484 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.277904 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.278052 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.280996 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:19.281002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:19.281007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:19.281043 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:19.284082 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.285661 (+ 1579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.285669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.285673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.285817 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.286171 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.286182 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.286185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.286450 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:19.286600 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.286960 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.287299 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":105}
I20260619 01:12:19.287592 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.273229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37134)
0619 01:12:19.273576 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:19.273597 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:19.273701 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:19.274041 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:19.274046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:19.274599 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:19.274771 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:19.274780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.275874 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.275879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:19.277347 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:19.277365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.278057 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.278062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:19.278264 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.278846 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:19.278865 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:19.280827 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:19.285116 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:19.285120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:19.285122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:19.285545 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:19.285925 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:19.285932 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:19.285937 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:19.286062 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:19.286564 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:19.286569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:19.286703 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 01:12:19.286859 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 01:12:19.287166 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260619 01:12:19.290740 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.276045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.0.97.61:43239)
0619 01:12:19.276202 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.276206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.276223 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:12:19.276405 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.276408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.276461 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.276534 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.278046 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.278955 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.280184 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.280686 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.280827 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.285153 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:19.285158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:19.285161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:19.285195 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:19.287995 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.288626 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.288632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.288635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.288792 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.289212 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.289217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.289221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.289578 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:19.289872 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.290148 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.290375 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:12:19.293080 23011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1: Bootstrap starting.
I20260619 01:12:19.296923 23011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:19.297998 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:12:19.298229 23011 log.cc:824] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:19.301177 23011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1: No bootstrap required, opened a new log
I20260619 01:12:19.300662 22838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f1108f07f1044fd080da2f8bc73301cd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40294
I20260619 01:12:19.312029 23011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } }
I20260619 01:12:19.312460 23011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c69295cec64b58835800db72fe3ed1, State: Initialized, Role: FOLLOWER
I20260619 01:12:19.312976 23011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } }
I20260619 01:12:19.313903 23017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } } }
I20260619 01:12:19.314239 23017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:19.315194 23011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:19.321614 22863 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } attempt: 4
W20260619 01:12:19.324982 23028 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:12:19.325111 23028 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 01:12:19 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:12:19.322939 22944 master_runner.cc:186] Error getting master registration for 127.0.97.62:44293: OK, instance_id { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" instance_seqno: 1781831538182712 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:12:19.336656 22944 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 01:12:19.339054 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.322344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37142)
0619 01:12:19.322659 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:19.322678 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:19.322780 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:19.323469 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:19.323474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:19.323987 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:19.324218 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:19.324231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.325812 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.325818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:19.326722 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:19.326734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.327840 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.327848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:19.328066 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.328728 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:19.328754 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:19.331773 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:19.336114 (+ 4341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:19.336120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:19.336125 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:19.336711 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:19.337148 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:19.337171 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:19.337174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:19.337387 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:19.338019 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:19.338024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:19.338198 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 01:12:19.338357 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 01:12:19.338665 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260619 01:12:19.339061 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.322393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.0.97.60:45965)
0619 01:12:19.322998 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.323007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.323027 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:12:19.323078 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.323082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.323159 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.323284 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.324423 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.325652 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.326875 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.327661 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.327785 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.331959 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:19.331972 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:19.331976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:19.332009 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:19.335948 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.336878 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.336884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.336888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.337050 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.337515 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.337520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.337524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.337893 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:19.338053 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.338451 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.338665 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478}
I20260619 01:12:19.345219 22863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd: Bootstrap starting.
I20260619 01:12:19.349689 22863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:19.351140 22863 log.cc:824] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:19.354924 22863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd: No bootstrap required, opened a new log
I20260619 01:12:19.363030 22863 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } }
I20260619 01:12:19.363514 22863 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1108f07f1044fd080da2f8bc73301cd, State: Initialized, Role: FOLLOWER
I20260619 01:12:19.364091 22863 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } }
I20260619 01:12:19.365175 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } } }
I20260619 01:12:19.365648 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:19.366567 22863 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:19.366631 22838 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 01:12:19.366888 22838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:19.369877 22838 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } }
I20260619 01:12:19.372098 22986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1108f07f1044fd080da2f8bc73301cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "58c69295cec64b58835800db72fe3ed1"
I20260619 01:12:19.372661 22986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:19.376283 22986 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1108f07f1044fd080da2f8bc73301cd in term 1.
I20260619 01:12:19.376721 22838 leader_election.cc:288] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [CANDIDATE]: Term 1 election: Requested vote from peers c465347475514503a9c93ab9d303edf3 (127.0.97.61:43239), 58c69295cec64b58835800db72fe3ed1 (127.0.97.60:45965)
I20260619 01:12:19.377059 22812 leader_election.cc:302] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58c69295cec64b58835800db72fe3ed1, f1108f07f1044fd080da2f8bc73301cd; no voters:
I20260619 01:12:19.377911 23031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:12:19.380021 23031 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [term 1 LEADER]: Becoming Leader. State: Replica: f1108f07f1044fd080da2f8bc73301cd, State: Running, Role: LEADER
I20260619 01:12:19.380553 23031 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } }
I20260619 01:12:19.386430 23037 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1108f07f1044fd080da2f8bc73301cd. Latest consensus state: current_term: 1 leader_uuid: "f1108f07f1044fd080da2f8bc73301cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } } }
I20260619 01:12:19.386878 23037 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:19.389891 23042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:12:19.416116 23042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:12:19.420610 23052 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 01:12:19.422191 22817 tablet.cc:2369] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:12:19.433486 22937 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } attempt: 5
W20260619 01:12:19.454684 22965 tablet.cc:2369] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:12:19.430130 22796 master_runner.cc:186] Error getting master registration for 127.0.97.61:43239: OK, instance_id { permanent_uuid: "c465347475514503a9c93ab9d303edf3" instance_seqno: 1781831538699719 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:12:19.460882 22796 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 01:12:19.466696 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.415387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.0.97.62:44293)
0619 01:12:19.415580 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.415587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.428249 (+ 12662us) server_negotiation.cc:407] Connection header received
0619 01:12:19.428792 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.428800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.428882 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.428964 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.431311 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.432536 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.434649 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.435496 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.435605 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.439337 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:19.439348 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:19.439356 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:19.439408 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:19.451917 (+ 12509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.458154 (+ 6237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.458161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.458165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.458381 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.458849 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.458855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.458858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.459228 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:19.459482 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.465959 (+ 6477us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.466272 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 01:12:19.474628 22986 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [term 1 FOLLOWER]: Refusing update from remote peer f1108f07f1044fd080da2f8bc73301cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:12:19.476341 23031 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Jun 19 01:12:19 dist-test-slave-5p8w krb5kdc[22783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:12:19.486673 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.423538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.0.97.61:43239)
0619 01:12:19.423740 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.423746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.447968 (+ 24222us) server_negotiation.cc:407] Connection header received
0619 01:12:19.448135 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.448143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.448244 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.448380 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.452638 (+ 4258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.454136 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.465151 (+ 11015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.466164 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.466324 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.471618 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:19.471624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:19.471627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:19.471668 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:19.482607 (+ 10939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.483567 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.483573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.483576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.483796 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.484334 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.484342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.484347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.484961 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:19.485240 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.485945 (+ 705us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.486204 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:12:19.496344 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.415989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.0.97.60:45965)
0619 01:12:19.416165 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.416171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.422983 (+ 6812us) server_negotiation.cc:407] Connection header received
0619 01:12:19.424466 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.424473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.424582 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.424660 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.447604 (+ 22944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.448706 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.472476 (+ 23770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.473235 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.473398 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.487843 (+ 14445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:19.487850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:19.487854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:19.487943 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:19.492236 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.493357 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.493363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.493367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.493552 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.494029 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.494035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.494041 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.494432 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:19.494678 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.495675 (+ 997us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.495945 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":12672}
I20260619 01:12:19.506606 23017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1108f07f1044fd080da2f8bc73301cd. Latest consensus state: current_term: 1 leader_uuid: "f1108f07f1044fd080da2f8bc73301cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } } }
I20260619 01:12:19.506968 23017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:19.507108 22940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.437627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45965 (local address 127.0.0.1:37160)
0619 01:12:19.438011 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:19.438030 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:19.438137 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:19.446395 (+ 8258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:19.446405 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:19.447105 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:19.447397 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:19.447414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.477276 (+ 29862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.477283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:19.478988 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:19.479008 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.483310 (+ 4302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.483317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:19.483557 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.484361 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:19.484456 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:19.487620 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:19.503269 (+ 15649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:19.503277 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:19.503281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:19.503914 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:19.504681 (+ 767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:19.504687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:19.504691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:19.504894 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:19.506023 (+ 1129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:19.506030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:19.506201 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 01:12:19.506364 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 01:12:19.506703 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"spinlock_wait_cycles":32128}
I20260619 01:12:19.507707 23054 mvcc.cc:205] Tried to move back new op lower bound from 7298381985675386880 to 7298381985307860992. Current Snapshot: MvccSnapshot[applied={T|T < 7298381985675386880}]
I20260619 01:12:19.510004 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1108f07f1044fd080da2f8bc73301cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } } }
I20260619 01:12:19.510329 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:19.513509 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.438472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.0.97.60:45965)
0619 01:12:19.445866 (+ 7394us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.445875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.445903 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:12:19.445967 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.445972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.446057 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.446181 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.474273 (+ 28092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.475635 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.479257 (+ 3622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.479980 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.487821 (+ 7841us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.489281 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:19.489330 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:19.489337 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:19.489373 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:19.502999 (+ 13626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.504152 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.504157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.504187 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.504474 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:19.505086 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:19.505091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:19.505095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:19.505821 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:19.507527 (+ 1706us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.507772 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.508580 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7249,"thread_start_us":6992,"threads_started":1}
I20260619 01:12:19.513875 23056 mvcc.cc:205] Tried to move back new op lower bound from 7298381985675386880 to 7298381985307860992. Current Snapshot: MvccSnapshot[applied={T|T < 7298381985675386880}]
I20260619 01:12:19.515973 23017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1108f07f1044fd080da2f8bc73301cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } } }
I20260619 01:12:19.516232 23017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:19.517055 23042 catalog_manager.cc:1347] Generated new cluster ID: 6781f8cb2bb34496853b7789c808f2b6
I20260619 01:12:19.517149 23042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:12:19.519500 22937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3: Bootstrap starting.
I20260619 01:12:19.526153 22937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:19.527658 22937 log.cc:824] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:19.543769 22937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3: No bootstrap required, opened a new log
I20260619 01:12:19.564777 22937 raft_consensus.cc:348] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } }
I20260619 01:12:19.565275 22937 raft_consensus.cc:729] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c465347475514503a9c93ab9d303edf3, State: Initialized, Role: FOLLOWER
I20260619 01:12:19.565898 22937 consensus_queue.cc:260] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } }
I20260619 01:12:19.566852 23042 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:12:19.567480 23061 sys_catalog.cc:455] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } } }
I20260619 01:12:19.568329 23061 sys_catalog.cc:458] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:19.568549 22937 sys_catalog.cc:564] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:19.568817 23042 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:12:19.568575 22912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1108f07f1044fd080da2f8bc73301cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c465347475514503a9c93ab9d303edf3"
I20260619 01:12:19.568955 22911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:19.572746 22911 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [term 1 FOLLOWER]: Refusing update from remote peer f1108f07f1044fd080da2f8bc73301cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:12:19.577332 23031 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:12:19.582450 22912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1108f07f1044fd080da2f8bc73301cd in term 1.
I20260619 01:12:19.597486 23042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd: Generated new TSK 0
I20260619 01:12:19.598673 23042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:12:19.620469 23068 sys_catalog.cc:455] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1108f07f1044fd080da2f8bc73301cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } } }
I20260619 01:12:19.620875 23068 sys_catalog.cc:458] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:19.621353 23037 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1108f07f1044fd080da2f8bc73301cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } } }
I20260619 01:12:19.621692 23037 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1108f07f1044fd080da2f8bc73301cd [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:19.622210 23061 sys_catalog.cc:455] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1108f07f1044fd080da2f8bc73301cd. Latest consensus state: current_term: 1 leader_uuid: "f1108f07f1044fd080da2f8bc73301cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1108f07f1044fd080da2f8bc73301cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44293 } } peers { permanent_uuid: "c465347475514503a9c93ab9d303edf3" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43239 } } peers { permanent_uuid: "58c69295cec64b58835800db72fe3ed1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45965 } } }
I20260619 01:12:19.622578 23061 sys_catalog.cc:458] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:19.636590 23077 catalog_manager.cc:1259] Loaded cluster ID: 6781f8cb2bb34496853b7789c808f2b6
I20260619 01:12:19.636806 23077 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3: loading cluster ID for follower catalog manager: success
I20260619 01:12:19.640852 23077 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3: acquiring CA information for follower catalog manager: success
I20260619 01:12:19.643513 23077 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c465347475514503a9c93ab9d303edf3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 01:12:19.747424 22866 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:19.727449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.0.97.62:44293)
0619 01:12:19.727684 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.727694 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.745162 (+ 17468us) server_negotiation.cc:407] Connection header received
0619 01:12:19.745734 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.745741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.745869 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.745948 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.747161 (+ 1213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40308: BlockingRecv error: recv got EOF from 127.0.0.1:40308 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 01:12:19.751951 22941 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:19.749238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.0.97.61:43239)
0619 01:12:19.749455 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.749463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.750081 (+ 618us) server_negotiation.cc:407] Connection header received
0619 01:12:19.750338 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.750351 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.750494 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.750565 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.751674 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39842: BlockingRecv error: recv got EOF from 127.0.0.1:39842 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260619 01:12:19.753993 23055 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:19.729696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.0.97.60:45965)
0619 01:12:19.729894 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.729901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.748301 (+ 18400us) server_negotiation.cc:407] Connection header received
0619 01:12:19.748459 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.748465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.748563 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.748687 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.753743 (+ 5056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37168: BlockingRecv error: recv got EOF from 127.0.0.1:37168 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:12:19.808593 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.789425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.0.97.61:43239)
0619 01:12:19.789630 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.789636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.800433 (+ 10797us) server_negotiation.cc:407] Connection header received
0619 01:12:19.800833 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.800839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.801091 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.801248 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:19.802033 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.803221 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.805405 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.806113 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.806226 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.807012 (+ 786us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:19.807743 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:19.807874 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.808108 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.808230 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 01:12:19.858340 23055 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:19.845058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.0.97.60:45965)
0619 01:12:19.845263 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.845271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.856003 (+ 10732us) server_negotiation.cc:407] Connection header received
0619 01:12:19.856578 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.856585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.856724 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.856810 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.858014 (+ 1204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37184: BlockingRecv error: recv got EOF from 127.0.0.1:37184 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:12:19.910455 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.895567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.0.97.61:43239)
0619 01:12:19.895790 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.895797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.903285 (+ 7488us) server_negotiation.cc:407] Connection header received
0619 01:12:19.903818 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.903825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.903924 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.903997 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:19.904688 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.905913 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.907984 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:19.908718 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:19.908818 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:19.909493 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:19.909701 (+ 208us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:19.909767 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:19.910036 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 01:12:19.910125 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 01:12:19.962720 23055 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:19.949536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.0.97.60:45965)
0619 01:12:19.949750 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:19.949757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:19.960534 (+ 10777us) server_negotiation.cc:407] Connection header received
0619 01:12:19.961074 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:19.961081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:19.961190 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:19.961308 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:19.962470 (+ 1162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37186: BlockingRecv error: recv got EOF from 127.0.0.1:37186 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:12:20.015118 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:19.999952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.0.97.61:43239)
0619 01:12:20.000140 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.000147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.008409 (+ 8262us) server_negotiation.cc:407] Connection header received
0619 01:12:20.008599 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.008605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.008724 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.008862 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:20.009395 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.010660 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.012791 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.013545 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.013705 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:20.014191 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:20.014375 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:20.014490 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:20.014669 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 01:12:20.014761 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 01:12:20.027554 23055 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:20.022302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.0.97.60:45965)
0619 01:12:20.022515 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.022522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.025939 (+ 3417us) server_negotiation.cc:407] Connection header received
0619 01:12:20.026157 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.026163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.026278 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.026393 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:20.027298 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37194: BlockingRecv error: recv got EOF from 127.0.0.1:37194 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 01:12:20.043870 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:20.037010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.0.97.61:43239)
0619 01:12:20.037275 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.037281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.037330 (+ 49us) server_negotiation.cc:407] Connection header received
0619 01:12:20.037685 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.037693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.037780 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.037878 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:20.038443 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.039641 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.041759 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.042429 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.042549 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:20.043074 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:20.043202 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:20.043300 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:20.043493 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 01:12:20.043575 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260619 01:12:20.059721 23055 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:20.051396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.0.97.60:45965)
0619 01:12:20.051645 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.051652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.057989 (+ 6337us) server_negotiation.cc:407] Connection header received
0619 01:12:20.058221 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.058228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.058335 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.058461 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:20.059457 (+ 996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37210: BlockingRecv error: recv got EOF from 127.0.0.1:37210 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260619 01:12:20.082720 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:20.075203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.0.97.61:43239)
0619 01:12:20.075423 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.075435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.076030 (+ 595us) server_negotiation.cc:407] Connection header received
0619 01:12:20.076230 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.076236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.076354 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.076467 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:20.076976 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.078175 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.080408 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.081114 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.081254 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:20.081832 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:20.081987 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:20.082076 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:20.082253 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 01:12:20.082354 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 01:12:20.093180 23055 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:20.091068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37220 (local address 127.0.97.60:45965)
0619 01:12:20.091270 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.091277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.091730 (+ 453us) server_negotiation.cc:407] Connection header received
0619 01:12:20.091939 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.091945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.092040 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.092155 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:20.092973 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37220: BlockingRecv error: recv got EOF from 127.0.0.1:37220 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:12:20.120152 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:20.113016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.0.97.61:43239)
0619 01:12:20.113200 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.113206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.113745 (+ 539us) server_negotiation.cc:407] Connection header received
0619 01:12:20.113941 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.113947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.114039 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.114154 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:20.114649 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.115913 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.117987 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.118674 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.118823 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:20.119347 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:20.119499 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:20.119596 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:20.119776 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 01:12:20.119875 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 01:12:20.128640 23055 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:20.126685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.0.97.60:45965)
0619 01:12:20.126865 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.126870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.127252 (+ 382us) server_negotiation.cc:407] Connection header received
0619 01:12:20.127476 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.127482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.127565 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.127669 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:20.128410 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37232: BlockingRecv error: recv got EOF from 127.0.0.1:37232 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 01:12:20.170701 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:20.164252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39910 (local address 127.0.97.61:43239)
0619 01:12:20.164487 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.164493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.164841 (+ 348us) server_negotiation.cc:407] Connection header received
0619 01:12:20.165008 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.165014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.165114 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.165250 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:20.165696 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.166880 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.168462 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.169190 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.169450 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:20.169851 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:20.170008 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:20.170109 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:20.170261 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 01:12:20.170359 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260619 01:12:20.179337 23055 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:20.177006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.0.97.60:45965)
0619 01:12:20.177283 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.177309 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.177651 (+ 342us) server_negotiation.cc:407] Connection header received
0619 01:12:20.177935 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.177940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.178016 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.178096 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:20.179066 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37236: BlockingRecv error: recv got EOF from 127.0.0.1:37236 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 01:12:20.252100 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:20.246858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.0.97.61:43239)
0619 01:12:20.247104 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.247111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.247383 (+ 272us) server_negotiation.cc:407] Connection header received
0619 01:12:20.247548 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.247552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.247622 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.247728 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:20.248181 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.248923 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.250452 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.250902 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.251010 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:20.251387 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:20.251497 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:20.251569 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:20.251705 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 01:12:20.251780 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 01:12:20.259449 23055 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:20.257711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.0.97.60:45965)
0619 01:12:20.257965 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.257972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.258167 (+ 195us) server_negotiation.cc:407] Connection header received
0619 01:12:20.258328 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.258332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.258406 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.258504 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:20.259238 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37250: BlockingRecv error: recv got EOF from 127.0.0.1:37250 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 01:12:20.327078 23028 catalog_manager.cc:1259] Loaded cluster ID: 6781f8cb2bb34496853b7789c808f2b6
I20260619 01:12:20.327175 23028 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1: loading cluster ID for follower catalog manager: success
I20260619 01:12:20.329653 23028 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1: acquiring CA information for follower catalog manager: success
I20260619 01:12:20.331418 23028 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 58c69295cec64b58835800db72fe3ed1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 01:12:20.399565 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:20.394172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39936 (local address 127.0.97.61:43239)
0619 01:12:20.394412 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.394418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.394749 (+ 331us) server_negotiation.cc:407] Connection header received
0619 01:12:20.394956 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.394961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.395054 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.395164 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:20.395694 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.396408 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.397817 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.398287 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.398434 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:20.398845 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:20.398952 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:20.399054 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:20.399204 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 01:12:20.399297 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260619 01:12:20.411249 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:20.405279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.0.97.60:45965)
0619 01:12:20.405526 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.405535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.405885 (+ 350us) server_negotiation.cc:407] Connection header received
0619 01:12:20.406100 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.406105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.406258 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.406395 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:20.406818 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.407512 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.408826 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.409281 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.409483 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:20.409998 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:20.410635 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:20.410816 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:20.410871 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 01:12:20.410957 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 01:12:20.425926 22941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:20.419119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.0.97.61:43239)
0619 01:12:20.419309 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.419314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.419918 (+ 604us) server_negotiation.cc:407] Connection header received
0619 01:12:20.420126 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.420131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.420205 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.420310 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:20.420901 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.421973 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.423674 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.424328 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.424464 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:20.425078 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:20.425229 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:20.425341 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:20.425534 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 01:12:20.425628 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:12:20.425926 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:20.418168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.0.97.60:45965)
0619 01:12:20.418414 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.418422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.419076 (+ 654us) server_negotiation.cc:407] Connection header received
0619 01:12:20.419381 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.419389 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.419499 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.419565 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:20.420508 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.421483 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.423877 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.424297 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.424369 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:20.425085 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:20.425232 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:20.425349 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:20.425578 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 01:12:20.425655 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:12:20.426635 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:20.417683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.0.97.62:44293)
0619 01:12:20.418200 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:20.418210 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:20.418872 (+ 662us) server_negotiation.cc:407] Connection header received
0619 01:12:20.419154 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:20.419160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:20.419384 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:20.419563 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:20.420503 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.421588 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.423833 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:20.424491 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:20.424644 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:20.425206 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:20.425699 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:20.425844 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:20.426202 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 01:12:20.426297 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":766,"threads_started":1}
I20260619 01:12:20.433943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22796
I20260619 01:12:20.435024 22858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:12:20.545394 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 22796
I20260619 01:12:20.566584 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22870
I20260619 01:12:20.567637 22932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:12:20.679522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 22870
I20260619 01:12:20.700314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 22944
I20260619 01:12:20.701468 23006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:12:20.810117 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 22944
2026-06-19T01:12:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4430 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:12:20 dist-test-slave-5p8w krb5kdc[23190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:12:20 dist-test-slave-5p8w krb5kdc[23190](info): set up 2 sockets
Jun 19 01:12:20 dist-test-slave-5p8w krb5kdc[23190](info): commencing operation
krb5kdc: starting...
W20260619 01:12:21.933957 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/test-user.keytab.
Jun 19 01:12:22 dist-test-slave-5p8w krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:12:22.202304 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32779
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32779,127.0.97.61:41885,127.0.97.60:41227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:12:22.544292 23203 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:12:22.544699 23203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:22.544827 23203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:22.544996 23203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:22.552865 23203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:22.552984 23203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:22.553099 23203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:22.553207 23203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:22.553333 23203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:22.563745 23203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32779,127.0.97.61:41885,127.0.97.60:41227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:22.565197 23203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:22.566692 23203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:22.579372 23211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:22.579404 23209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:22.579334 23208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:22.579993 23203 server_base.cc:1034] running on GCE node
I20260619 01:12:22.581010 23203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:22.582732 23203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:22.583962 23203 hybrid_clock.cc:648] HybridClock initialized: now 1781831542583898 us; error 30 us; skew 500 ppm
I20260619 01:12:22.584323 23203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:22 dist-test-slave-5p8w krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:22.587774 23203 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:12:22.589221 23203 webserver.cc:460] Webserver started at http://127.0.97.62:44315/ using document root <none> and password file <none>
I20260619 01:12:22.589967 23203 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:22.590055 23203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:22.590330 23203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:22.593513 23203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/data/instance:
uuid: "3544a17c2ac14fd1935c2f8f450bcf8f"
format_stamp: "Formatted at 2026-06-19 01:12:22 on dist-test-slave-5p8w"
server_key: "462cdd9d637066f34ee83839792aa381"
server_key_iv: "30026df2293c405747f54f47351c93f3"
server_key_version: "encryptionkey@0"
I20260619 01:12:22.594586 23203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/wal/instance:
uuid: "3544a17c2ac14fd1935c2f8f450bcf8f"
format_stamp: "Formatted at 2026-06-19 01:12:22 on dist-test-slave-5p8w"
server_key: "462cdd9d637066f34ee83839792aa381"
server_key_iv: "30026df2293c405747f54f47351c93f3"
server_key_version: "encryptionkey@0"
I20260619 01:12:22.600852 23203 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 01:12:22.604470 23218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:22.606019 23203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:12:22.606215 23203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/wal
uuid: "3544a17c2ac14fd1935c2f8f450bcf8f"
format_stamp: "Formatted at 2026-06-19 01:12:22 on dist-test-slave-5p8w"
server_key: "462cdd9d637066f34ee83839792aa381"
server_key_iv: "30026df2293c405747f54f47351c93f3"
server_key_version: "encryptionkey@0"
I20260619 01:12:22.606451 23203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:22.623075 23203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:22.627758 23203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:22.628072 23203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:22.648989 23203 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32779
I20260619 01:12:22.649008 23269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32779 every 8 connection(s)
I20260619 01:12:22.650405 23203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:12:22.654206 23270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:22.658769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 23203
I20260619 01:12:22.658949 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-0/wal/instance
I20260619 01:12:22.659490 388 external_mini_cluster.cc:1428] Setting key 6c06f7b7495a4cd964c21213530089ab
I20260619 01:12:22.661882 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:12:22 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260619 01:12:22.688521 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:22.665954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.0.97.62:32779)
0619 01:12:22.668000 (+ 2046us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:22.668007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:22.668052 (+ 45us) server_negotiation.cc:407] Connection header received
0619 01:12:22.670315 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:22.670324 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:22.670688 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:22.670947 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:22.672085 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:22.674130 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:22.675621 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:22.676549 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:22.676738 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:22.680072 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:22.680085 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:22.680106 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:22.680147 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:22.684175 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:22.684902 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:22.684913 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:22.684920 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:22.685127 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:22.685584 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:22.685591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:22.685595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:22.686344 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:22.686567 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:22.687143 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 01:12:22.687525 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":1853,"spinlock_wait_cycles":30592,"thread_start_us":1609,"threads_started":1}
I20260619 01:12:22.688560 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:22.665954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32779 (local address 127.0.0.1:42170)
0619 01:12:22.666977 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:22.667035 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:22.669905 (+ 2870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:22.670966 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:22.670972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:22.671533 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:22.671872 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:22.671889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:22.674280 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:22.674284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:22.675477 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:22.675489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:22.676760 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:22.676764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:22.676907 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:22.677646 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:22.677668 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:22.679829 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:22.684330 (+ 4501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:22.684341 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:22.684352 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:22.684747 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:22.685253 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:22.685256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:22.685258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:22.685473 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:22.686488 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:22.686494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:22.686690 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 01:12:22.686910 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 01:12:22.687321 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":452,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 01:12:22.691727 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:12:22.693105 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:22.692517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55360)
0619 01:12:22.692788 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:22.692931 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 01:12:22.693509 23219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
I20260619 01:12:22.695266 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:22.694784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55372)
0619 01:12:22.694979 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:22.695071 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 01:12:22.695464 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41885: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 01:12:22.722618 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41885
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32779,127.0.97.61:41885,127.0.97.60:41227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:12:22.744848 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } attempt: 1
I20260619 01:12:22.746150 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:22.745722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55380)
0619 01:12:22.745917 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:22.746006 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 01:12:22.747215 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:22.746866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55382)
0619 01:12:22.747023 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:22.747090 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 01:12:22.747404 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41885: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
I20260619 01:12:22.815649 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } attempt: 2
I20260619 01:12:22.816812 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:22.816319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55390)
0619 01:12:22.816577 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:22.816667 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 01:12:22.817884 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:22.817517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55406)
0619 01:12:22.817669 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:22.817739 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 01:12:22.818056 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41885: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
I20260619 01:12:22.909358 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } attempt: 3
I20260619 01:12:22.910483 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:22.910036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55422)
0619 01:12:22.910240 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:22.910350 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 01:12:22.911530 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:22.911146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55432)
0619 01:12:22.911289 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:22.911387 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 01:12:22.911710 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41885: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
I20260619 01:12:23.054941 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } attempt: 4
I20260619 01:12:23.056092 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.055564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55436)
0619 01:12:23.055862 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.055961 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260619 01:12:23.057152 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.056806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55446)
0619 01:12:23.056955 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.057031 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 01:12:23.057366 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41885: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41885: connect: Connection refused (error 111)
W20260619 01:12:23.058516 23277 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:12:23.058884 23277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:23.059008 23277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:23.059141 23277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:12:23.066576 23277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:23.066711 23277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:23.066838 23277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:23.066924 23277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:23.067032 23277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:23.077534 23277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32779,127.0.97.61:41885,127.0.97.60:41227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:23.078940 23277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:23.080350 23277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:23.092214 23282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:23.092152 23283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:23.092168 23285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:23.093214 23277 server_base.cc:1034] running on GCE node
I20260619 01:12:23.094175 23277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:23.095657 23277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:23.096915 23277 hybrid_clock.cc:648] HybridClock initialized: now 1781831543096848 us; error 46 us; skew 500 ppm
I20260619 01:12:23.097254 23277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:23.100385 23277 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 01:12:23.101902 23277 webserver.cc:460] Webserver started at http://127.0.97.61:41815/ using document root <none> and password file <none>
I20260619 01:12:23.102615 23277 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:23.102758 23277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:23.103081 23277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:23.106314 23277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/data/instance:
uuid: "217f7bc0df244612b2bb21f453ede193"
format_stamp: "Formatted at 2026-06-19 01:12:23 on dist-test-slave-5p8w"
server_key: "c79537a13145c8445d04e5554624d13c"
server_key_iv: "7d878860bc936b9956989a379f75fbd3"
server_key_version: "encryptionkey@0"
I20260619 01:12:23.107117 23277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/wal/instance:
uuid: "217f7bc0df244612b2bb21f453ede193"
format_stamp: "Formatted at 2026-06-19 01:12:23 on dist-test-slave-5p8w"
server_key: "c79537a13145c8445d04e5554624d13c"
server_key_iv: "7d878860bc936b9956989a379f75fbd3"
server_key_version: "encryptionkey@0"
I20260619 01:12:23.112394 23277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260619 01:12:23.115916 23292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:23.117255 23277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:12:23.117496 23277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/wal
uuid: "217f7bc0df244612b2bb21f453ede193"
format_stamp: "Formatted at 2026-06-19 01:12:23 on dist-test-slave-5p8w"
server_key: "c79537a13145c8445d04e5554624d13c"
server_key_iv: "7d878860bc936b9956989a379f75fbd3"
server_key_version: "encryptionkey@0"
I20260619 01:12:23.117738 23277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:23.144482 23277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:23.148610 23277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:23.148902 23277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:23.167768 23277 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41885
I20260619 01:12:23.167802 23343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41885 every 8 connection(s)
I20260619 01:12:23.169224 23277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/data/info.pb
I20260619 01:12:23.173097 23344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:23.178781 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 23277
I20260619 01:12:23.178960 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-1/wal/instance
I20260619 01:12:23.179489 388 external_mini_cluster.cc:1428] Setting key edbf1d8b1b6fe26e772ecf7f6c0efb16
I20260619 01:12:23.180289 23344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:12:23.203575 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.183911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.0.97.62:32779)
0619 01:12:23.184139 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.184147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.185432 (+ 1285us) server_negotiation.cc:407] Connection header received
0619 01:12:23.187365 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.187374 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.187476 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.187599 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.189178 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.189948 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.192442 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.192884 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.192981 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.197264 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.197269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.197273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.197358 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.200054 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.200823 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.200829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.200834 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.200958 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.201447 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.201456 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.201463 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.201898 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.202082 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.202817 (+ 735us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.203106 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 01:12:23.204175 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.184343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32779 (local address 127.0.0.1:42176)
0619 01:12:23.185218 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.185282 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:23.187075 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:23.187821 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:23.187829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:23.188378 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:23.188971 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.188989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.190136 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.190144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.192297 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.192315 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.193121 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.193130 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.193496 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.194309 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:23.194343 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:23.197016 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:23.200217 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.200225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.200244 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.200676 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.201072 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.201077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.201081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.201277 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.202031 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:23.202038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:23.202392 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 01:12:23.202619 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 01:12:23.203026 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":387,"threads_started":1}
I20260619 01:12:23.206302 23344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:12:23.222330 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.207139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55458)
0619 01:12:23.207335 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.207352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:23.207479 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:23.208514 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:23.208518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:23.208911 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:23.209111 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.209120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.210760 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.210769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.212291 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.212308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.213122 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.213130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.213372 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.213966 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:23.213993 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:23.216446 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:23.219593 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.219598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.219602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.219967 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.220368 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.220373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.220376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.220521 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.221209 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:23.221213 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:23.221408 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 01:12:23.221565 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 01:12:23.221852 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 01:12:23.222426 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.207396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.0.97.61:41885)
0619 01:12:23.208036 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.208042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.208068 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:12:23.208110 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.208115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.208371 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.208490 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.209284 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.210563 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.212426 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.212962 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.213062 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.216644 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.216654 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.216667 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.216698 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.219456 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.220094 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.220108 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.220111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.220219 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.220615 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.220622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.220626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.221076 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.221330 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.221783 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.221996 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":358,"threads_started":1}
I20260619 01:12:23.224764 23344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:12:23.226078 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.225488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34060)
0619 01:12:23.225692 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.225840 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 01:12:23.226403 23296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
I20260619 01:12:23.228067 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.227621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34076)
0619 01:12:23.227808 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.227913 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 01:12:23.228273 23344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 01:12:23.241513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbdbgje/build/asan/bin/kudu
/tmp/dist-test-taskbdbgje/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41227
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32779,127.0.97.61:41885,127.0.97.60:41227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:12:23.277637 23344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } attempt: 1
I20260619 01:12:23.278834 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.278379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34080)
0619 01:12:23.278586 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.278679 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 01:12:23.279975 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.279577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34096)
0619 01:12:23.279753 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.279861 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 01:12:23.280329 23344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
I20260619 01:12:23.348596 23344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } attempt: 2
I20260619 01:12:23.349869 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.349402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34104)
0619 01:12:23.349598 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.349708 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 01:12:23.350972 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.350542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34116)
0619 01:12:23.350712 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.350800 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 01:12:23.351177 23344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
I20260619 01:12:23.356612 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } attempt: 5
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:12:23.371044 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.357435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.0.97.61:41885)
0619 01:12:23.357608 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.357612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.357724 (+ 112us) server_negotiation.cc:407] Connection header received
0619 01:12:23.357937 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.357941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.358019 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.358106 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.359119 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.359810 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.361521 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.361969 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.362071 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.365690 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.365695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.365698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.365733 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.368367 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.369061 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.369071 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.369074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.369198 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.369643 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.369647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.369649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.369936 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.370094 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.370484 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.370712 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:12:23.371064 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.357302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55466)
0619 01:12:23.357599 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.357626 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:23.357802 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:23.358196 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:23.358202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:23.358729 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:23.358968 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.358980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.359967 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.359973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.361360 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.361375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.362125 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.362131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.362333 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.363002 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:23.363025 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:23.365460 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:23.368525 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.368532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.368537 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.368926 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.369370 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.369375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.369379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.369517 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.370056 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:23.370061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:23.370224 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 01:12:23.370374 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 01:12:23.370674 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 01:12:23.372153 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:12:23.373165 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.372790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34124)
0619 01:12:23.372951 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.373033 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 01:12:23.374253 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.373820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34140)
0619 01:12:23.374015 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.374101 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 01:12:23.374500 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
I20260619 01:12:23.425755 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } attempt: 1
I20260619 01:12:23.427000 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.426435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34144)
0619 01:12:23.426676 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.426791 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 01:12:23.428045 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.427670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34156)
0619 01:12:23.427820 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.427907 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 01:12:23.428259 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
I20260619 01:12:23.442415 23344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } attempt: 3
I20260619 01:12:23.443667 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.443152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34170)
0619 01:12:23.443389 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.443480 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 01:12:23.445003 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.444532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34178)
0619 01:12:23.444784 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.444873 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260619 01:12:23.445370 23344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
I20260619 01:12:23.496495 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } attempt: 2
I20260619 01:12:23.497773 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.497171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34180)
0619 01:12:23.497420 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.497532 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 01:12:23.498958 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.498493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34188)
0619 01:12:23.498670 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.498768 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 01:12:23.499205 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
W20260619 01:12:23.584621 23351 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:12:23.584965 23351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:12:23.585037 23351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:12:23.585119 23351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 01:12:23.588613 23344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } attempt: 4
I20260619 01:12:23.589967 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.589387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34204)
0619 01:12:23.589703 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.589802 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260619 01:12:23.591143 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.590757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34218)
0619 01:12:23.590951 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.591027 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 01:12:23.591398 23344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
W20260619 01:12:23.593119 23351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:12:23.593204 23351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:12:23.593371 23351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:12:23.593453 23351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:12:23.593554 23351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:12:23.605031 23351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32779,127.0.97.61:41885,127.0.97.60:41227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:12:23.605391 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } attempt: 3
I20260619 01:12:23.606592 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.606109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34220)
0619 01:12:23.606350 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.606434 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 01:12:23.606737 23351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:12:23.607636 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.607285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34234)
0619 01:12:23.607452 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.607535 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 01:12:23.607937 23270 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41227: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41227: connect: Connection refused (error 111)
I20260619 01:12:23.608402 23351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:12:23.620469 23357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:23.620496 23356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:12:23.620626 23359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:12:23.621271 23351 server_base.cc:1034] running on GCE node
I20260619 01:12:23.622224 23351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:12:23.623689 23351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:12:23.624961 23351 hybrid_clock.cc:648] HybridClock initialized: now 1781831543624886 us; error 38 us; skew 500 ppm
I20260619 01:12:23.625353 23351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:12:23.628646 23351 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 01:12:23.630223 23351 webserver.cc:460] Webserver started at http://127.0.97.60:33983/ using document root <none> and password file <none>
I20260619 01:12:23.631004 23351 fs_manager.cc:359] Metadata directory not provided
I20260619 01:12:23.631124 23351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:12:23.631466 23351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:12:23.634559 23351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/data/instance:
uuid: "842e4c132baf444d9c36a703a967e52d"
format_stamp: "Formatted at 2026-06-19 01:12:23 on dist-test-slave-5p8w"
server_key: "7e2fb7eff7fb5251aedfc0dbd6f4b26a"
server_key_iv: "65ca9e217031273fe403a56b90571380"
server_key_version: "encryptionkey@0"
I20260619 01:12:23.635371 23351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/wal/instance:
uuid: "842e4c132baf444d9c36a703a967e52d"
format_stamp: "Formatted at 2026-06-19 01:12:23 on dist-test-slave-5p8w"
server_key: "7e2fb7eff7fb5251aedfc0dbd6f4b26a"
server_key_iv: "65ca9e217031273fe403a56b90571380"
server_key_version: "encryptionkey@0"
I20260619 01:12:23.640687 23351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 01:12:23.644229 23366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:12:23.645697 23351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 01:12:23.645898 23351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/wal
uuid: "842e4c132baf444d9c36a703a967e52d"
format_stamp: "Formatted at 2026-06-19 01:12:23 on dist-test-slave-5p8w"
server_key: "7e2fb7eff7fb5251aedfc0dbd6f4b26a"
server_key_iv: "65ca9e217031273fe403a56b90571380"
server_key_version: "encryptionkey@0"
I20260619 01:12:23.646127 23351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:12:23.668453 23351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:12:23.672991 23351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:12:23.673283 23351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:12:23.692116 23351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41227
I20260619 01:12:23.692135 23417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41227 every 8 connection(s)
I20260619 01:12:23.693622 23351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/data/info.pb
I20260619 01:12:23.697476 23418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:12:23.697824 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbdbgje/build/asan/bin/kudu as pid 23351
I20260619 01:12:23.698000 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbdbgje/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781831171286417-388-0/minicluster-data/master-2/wal/instance
I20260619 01:12:23.698544 388 external_mini_cluster.cc:1428] Setting key 54059dc5ddd1787b84f5eaf1fcde9840
I20260619 01:12:23.706664 23418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:12:23.723764 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.700330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.0.97.60:41227)
0619 01:12:23.701161 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.701171 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.701233 (+ 62us) server_negotiation.cc:407] Connection header received
0619 01:12:23.703623 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.703644 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.704181 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.704518 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.706225 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.707568 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.709200 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.710409 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.710586 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.715514 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.715521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.715532 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.715565 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.719203 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.719969 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.719977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.719986 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.720165 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.720599 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.720609 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.720612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.721404 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.721689 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.722294 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.722743 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":440,"threads_started":1}
I20260619 01:12:23.725782 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.709397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.0.97.62:32779)
0619 01:12:23.709962 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.709969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.710543 (+ 574us) server_negotiation.cc:407] Connection header received
0619 01:12:23.710704 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.710711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.710797 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.710910 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.711989 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.713231 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.714899 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.715559 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.715652 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.719726 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.719732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.719736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.719769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.723021 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.723679 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.723685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.723689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.723861 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.724257 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.724262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.724266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.724518 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.724678 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.725128 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.725372 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":241,"threads_started":1}
I20260619 01:12:23.725813 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.709430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32779 (local address 127.0.0.1:42178)
0619 01:12:23.710360 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.710410 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:23.710572 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:23.710947 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:23.710952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:23.711589 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:23.711831 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.711839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.713411 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.713414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.714754 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.714766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.715744 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.715750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.715954 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.716742 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:23.716771 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:23.719536 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:23.723152 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.723164 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.723171 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.723563 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.724005 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.724009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.724013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.724136 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.724657 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:23.724665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:23.724849 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 01:12:23.725041 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 01:12:23.725350 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":462,"threads_started":1}
I20260619 01:12:23.727177 23418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:12:23.740443 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.727915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41885 (local address 127.0.0.1:55468)
0619 01:12:23.728231 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.728248 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:23.728345 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:23.728674 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:23.728679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:23.729027 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:23.729147 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.729154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.730110 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.730115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.730980 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.730990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.731686 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.731692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.731891 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.732395 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:23.732410 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:23.734581 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:23.738150 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.738154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.738156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.738507 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.738908 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.738913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.738917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.739079 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.739635 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:23.739640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:23.739806 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 01:12:23.739953 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 01:12:23.740146 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222}
I20260619 01:12:23.740978 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.728065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.0.97.61:41885)
0619 01:12:23.728222 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.728236 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.728295 (+ 59us) server_negotiation.cc:407] Connection header received
0619 01:12:23.728487 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.728491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.728549 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.728647 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.729318 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.729967 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.731118 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.731529 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.731686 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.734750 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.734755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.734758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.734789 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.738025 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.738618 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.738623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.738627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.738800 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.739183 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.739186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.739188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.739507 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.739658 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.740060 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.740289 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 01:12:23.741457 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.726810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.0.97.62:32779)
0619 01:12:23.726967 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.726974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.726992 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:12:23.727137 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.727144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.727242 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.727341 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.728507 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.729185 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.730432 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.730828 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.730958 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.735454 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.735462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.735469 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.735511 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.738624 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.739377 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.739382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.739385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.739508 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.739905 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.739908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.739910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.740251 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.740649 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.740819 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.741148 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 01:12:23.742274 23418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:12:23.756938 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.743123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34246)
0619 01:12:23.743334 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.743350 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:23.743462 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:23.743796 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:23.743800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:23.744284 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:23.744483 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.744496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.745575 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.745578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.746737 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.746752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.747848 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.747852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.748014 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.748596 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:23.748612 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:23.751129 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:23.754421 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.754425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.754428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.754871 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.755300 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.755303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.755306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.755455 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.756060 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:23.756063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:23.756181 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 01:12:23.756299 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 01:12:23.756521 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260619 01:12:23.756973 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.743041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.0.97.60:41227)
0619 01:12:23.743208 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.743213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.743430 (+ 217us) server_negotiation.cc:407] Connection header received
0619 01:12:23.743578 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.743583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.743651 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.743749 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.744635 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.745437 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.746919 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.747679 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.747813 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.751297 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.751317 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.751320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.751347 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.754267 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.754996 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.755005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.755008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.755160 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.755567 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.755572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.755575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.755946 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.756100 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.756410 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.756712 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":46}
I20260619 01:12:23.759397 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.743207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.0.97.61:41885)
0619 01:12:23.743376 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.743379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.743429 (+ 50us) server_negotiation.cc:407] Connection header received
0619 01:12:23.743587 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.743591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.743651 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.743757 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.745182 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.746005 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.747660 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.748196 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.748293 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.753282 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.753318 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.753323 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.753361 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.756555 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.757449 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.757455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.757459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.757622 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.757999 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.758006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.758010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.758341 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.758614 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.758824 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.759048 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:12:23.762490 23418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d: Bootstrap starting.
I20260619 01:12:23.764016 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:12:23.765959 23418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:23.766995 23418 log.cc:824] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:23.766276 23245 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42180
I20260619 01:12:23.769768 23418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d: No bootstrap required, opened a new log
I20260619 01:12:23.776798 23418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } }
I20260619 01:12:23.777137 23418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842e4c132baf444d9c36a703a967e52d, State: Initialized, Role: FOLLOWER
I20260619 01:12:23.777635 23418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } }
I20260619 01:12:23.778609 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } } }
I20260619 01:12:23.778995 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:23.779567 23418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:23.785164 23270 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } attempt: 4
W20260619 01:12:23.789212 23436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:12:23.789358 23436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:12:23.786948 23351 master_runner.cc:186] Error getting master registration for 127.0.97.62:32779: OK, instance_id { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" instance_seqno: 1781831542643595 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:12:23.798712 23351 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 01:12:23.801537 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.785884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34258)
0619 01:12:23.786138 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.786163 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:23.786289 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:23.786863 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:23.786868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:23.787364 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:23.787594 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.787606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.789199 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.789206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.790273 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.790285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.791241 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.791248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.791449 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.792108 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:23.792125 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:23.794590 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:23.798877 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.798882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.798887 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.799391 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.799791 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.799797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.799800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.799952 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.800504 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:23.800510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:23.800677 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 01:12:23.800821 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 01:12:23.801127 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260619 01:12:23.801534 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.785931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.0.97.60:41227)
0619 01:12:23.786123 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.786131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.786237 (+ 106us) server_negotiation.cc:407] Connection header received
0619 01:12:23.786437 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.786446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.786556 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.786683 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.787773 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.788997 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.790436 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.791063 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.791224 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.794775 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.794783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.794786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.794815 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.798672 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.799535 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.799541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.799545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.799677 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.800065 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.800071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.800074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.800383 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.800560 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.800918 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.801123 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:12:23.806402 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f: Bootstrap starting.
I20260619 01:12:23.809608 23270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:23.810751 23270 log.cc:824] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:23.813539 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f: No bootstrap required, opened a new log
I20260619 01:12:23.818909 23270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } }
I20260619 01:12:23.819247 23270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3544a17c2ac14fd1935c2f8f450bcf8f, State: Initialized, Role: FOLLOWER
I20260619 01:12:23.819693 23270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } }
I20260619 01:12:23.820919 23439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } } }
I20260619 01:12:23.821614 23439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:23.821768 23245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 01:12:23.822108 23245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:23.824198 23245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } }
I20260619 01:12:23.821702 23270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:23.827071 23245 leader_election.cc:288] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [CANDIDATE]: Term 1 election: Requested vote from peers 217f7bc0df244612b2bb21f453ede193 (127.0.97.61:41885), 842e4c132baf444d9c36a703a967e52d (127.0.97.60:41227)
I20260619 01:12:23.827893 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "842e4c132baf444d9c36a703a967e52d"
I20260619 01:12:23.828401 23393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:23.831933 23393 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3544a17c2ac14fd1935c2f8f450bcf8f in term 1.
I20260619 01:12:23.832617 23222 leader_election.cc:302] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3544a17c2ac14fd1935c2f8f450bcf8f, 842e4c132baf444d9c36a703a967e52d; no voters:
I20260619 01:12:23.833127 23439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:12:23.849608 23439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [term 1 LEADER]: Becoming Leader. State: Replica: 3544a17c2ac14fd1935c2f8f450bcf8f, State: Running, Role: LEADER
I20260619 01:12:23.850860 23439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } }
I20260619 01:12:23.862941 23445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3544a17c2ac14fd1935c2f8f450bcf8f. Latest consensus state: current_term: 1 leader_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } } }
I20260619 01:12:23.863350 23445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:23.864251 23455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:12:23.877898 23455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 01:12:23.881922 23224 tablet.cc:2369] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:12:23.890594 23344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } attempt: 5
I20260619 01:12:23.919765 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.862091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.0.97.62:32779)
0619 01:12:23.862328 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.862338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.865771 (+ 3433us) server_negotiation.cc:407] Connection header received
0619 01:12:23.866853 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.866862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.866967 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.867050 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.872682 (+ 5632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.873956 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.894143 (+ 20187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.895065 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.895192 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.906936 (+ 11744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.906944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.906948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.907038 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.914675 (+ 7637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.915872 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.915878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.915882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.916069 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.916791 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.916807 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.916810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.917249 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.917565 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.918525 (+ 960us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.918967 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 01:12:23.922758 23348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.864564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.0.97.61:41885)
0619 01:12:23.864770 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.864788 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.870101 (+ 5313us) server_negotiation.cc:407] Connection header received
0619 01:12:23.870435 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.870444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.870557 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.870651 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.890363 (+ 19712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.891647 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.899553 (+ 7906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.900426 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.900552 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.907005 (+ 6453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.907013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.907017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.907058 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.912851 (+ 5793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.920000 (+ 7149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.920010 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.920016 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.920222 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.920676 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.920694 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.920698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.921109 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.921390 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.921940 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.922286 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 01:12:23.925689 23372 tablet.cc:2369] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 01:12:23 dist-test-slave-5p8w krb5kdc[23190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:12:23.930248 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.871236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.0.97.60:41227)
0619 01:12:23.871456 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.871465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.889961 (+ 18496us) server_negotiation.cc:407] Connection header received
0619 01:12:23.890260 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.890266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.890353 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.890429 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.901121 (+ 10692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.902284 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.906914 (+ 4630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.907536 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.907635 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.922661 (+ 15026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.922668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.922674 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.922720 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.926768 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.927679 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.927686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.927689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.927845 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.928355 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.928361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.928366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.928701 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.928975 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.929616 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.929859 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 01:12:23.911285 23203 master_runner.cc:186] Error getting master registration for 127.0.97.61:41885: OK, instance_id { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" instance_seqno: 1781831543162387 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:12:23.931469 23203 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 01:12:23.946106 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.891215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41227 (local address 127.0.0.1:34278)
0619 01:12:23.891576 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 01:12:23.891595 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:12:23.891698 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:12:23.910097 (+ 18399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:12:23.910103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:12:23.910614 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:12:23.910852 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.910865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.920245 (+ 9380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.920252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.921907 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:12:23.921926 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.926612 (+ 4686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.926621 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:12:23.926852 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.927543 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:12:23.927568 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:12:23.930500 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:12:23.941248 (+ 10748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.941255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.941259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.943678 (+ 2419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.944145 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:12:23.944152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:12:23.944156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:12:23.944329 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:12:23.945018 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:12:23.945023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:12:23.945201 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 01:12:23.945370 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 01:12:23.945730 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260619 01:12:23.946159 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:23.891920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.0.97.60:41227)
0619 01:12:23.909630 (+ 17710us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:23.909636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:23.909660 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:12:23.909711 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:23.909716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:23.909826 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:23.913151 (+ 3325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:23.913470 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.914738 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.922125 (+ 7387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:23.924798 (+ 2673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:23.925967 (+ 1169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:23.930858 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:12:23.930869 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:12:23.930876 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:12:23.930940 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:12:23.941076 (+ 10136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.943797 (+ 2721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.943805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.943809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.943972 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:12:23.944491 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:12:23.944497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:12:23.944500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:12:23.944894 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:12:23.945059 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:23.945492 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 01:12:23.945721 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17571,"thread_start_us":17363,"threads_started":1}
I20260619 01:12:23.952632 23344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193: Bootstrap starting.
I20260619 01:12:23.958635 23344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193: Neither blocks nor log segments found. Creating new log.
I20260619 01:12:23.960042 23344 log.cc:824] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193: Log is configured to *not* fsync() on all Append() calls
I20260619 01:12:23.963763 23344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193: No bootstrap required, opened a new log
I20260619 01:12:23.967175 23393 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [term 1 FOLLOWER]: Refusing update from remote peer 3544a17c2ac14fd1935c2f8f450bcf8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:12:23.968374 23445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:12:23.975549 23344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } }
I20260619 01:12:23.976008 23344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217f7bc0df244612b2bb21f453ede193, State: Initialized, Role: FOLLOWER
I20260619 01:12:23.976577 23344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } }
I20260619 01:12:23.996351 23344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:12:23.996399 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "217f7bc0df244612b2bb21f453ede193"
I20260619 01:12:23.999565 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3544a17c2ac14fd1935c2f8f450bcf8f. Latest consensus state: current_term: 1 leader_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } } }
I20260619 01:12:23.999894 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:24.001281 23318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:12:24.005657 23318 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [term 1 FOLLOWER]: Refusing update from remote peer 3544a17c2ac14fd1935c2f8f450bcf8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:12:24.010489 23319 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3544a17c2ac14fd1935c2f8f450bcf8f in term 1.
I20260619 01:12:24.015861 23445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:12:24.016263 23465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } } }
I20260619 01:12:24.016813 23465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:24.019568 23463 mvcc.cc:205] Tried to move back new op lower bound from 7298382004029353984 to 7298382003620261888. Current Snapshot: MvccSnapshot[applied={T|T < 7298382004029353984}]
I20260619 01:12:24.034752 23466 mvcc.cc:205] Tried to move back new op lower bound from 7298382004029353984 to 7298382003620261888. Current Snapshot: MvccSnapshot[applied={T|T < 7298382004029353984}]
I20260619 01:12:24.044716 23465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3544a17c2ac14fd1935c2f8f450bcf8f. Latest consensus state: current_term: 1 leader_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } } }
I20260619 01:12:24.045223 23465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:24.046808 23439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } } }
I20260619 01:12:24.047156 23439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:24.048321 23439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } } }
I20260619 01:12:24.048712 23439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f [sys.catalog]: This master's current role is: LEADER
I20260619 01:12:24.050596 23455 catalog_manager.cc:1347] Generated new cluster ID: 4c493cb25c1c45ecba625346b813f4b8
I20260619 01:12:24.050729 23455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:12:24.065663 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } } }
I20260619 01:12:24.067160 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:24.081779 23465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3544a17c2ac14fd1935c2f8f450bcf8f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32779 } } peers { permanent_uuid: "217f7bc0df244612b2bb21f453ede193" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41885 } } peers { permanent_uuid: "842e4c132baf444d9c36a703a967e52d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41227 } } }
I20260619 01:12:24.082167 23465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:12:24.095614 23483 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 01:12:24.098819 23455 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:12:24.100709 23455 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:12:24.119055 23455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3544a17c2ac14fd1935c2f8f450bcf8f: Generated new TSK 0
I20260619 01:12:24.120293 23455 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 01:12:24.221689 23348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.212617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.0.97.61:41885)
0619 01:12:24.213090 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.213100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.219054 (+ 5954us) server_negotiation.cc:407] Connection header received
0619 01:12:24.219409 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.219419 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.219544 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.219640 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.221328 (+ 1688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55514: BlockingRecv error: recv got EOF from 127.0.0.1:55514 (error 108)
Metrics: {"server-negotiator.queue_time_us":301}
W20260619 01:12:24.224926 23462 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.218512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.0.97.60:41227)
0619 01:12:24.218765 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.218773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.222747 (+ 3974us) server_negotiation.cc:407] Connection header received
0619 01:12:24.223048 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.223062 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.223204 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.223301 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.224569 (+ 1268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34288: BlockingRecv error: recv got EOF from 127.0.0.1:34288 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 01:12:24.227414 23422 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.212844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.0.97.62:32779)
0619 01:12:24.213355 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.213364 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.217909 (+ 4545us) server_negotiation.cc:407] Connection header received
0619 01:12:24.226083 (+ 8174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.226091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.226182 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.226294 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.227153 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42196: BlockingRecv error: recv got EOF from 127.0.0.1:42196 (error 108)
Metrics: {"server-negotiator.queue_time_us":283}
W20260619 01:12:24.235324 23348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.231078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55526 (local address 127.0.97.61:41885)
0619 01:12:24.231328 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.231337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.233858 (+ 2521us) server_negotiation.cc:407] Connection header received
0619 01:12:24.234101 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.234105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.234225 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.234343 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.235058 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55526: BlockingRecv error: recv got EOF from 127.0.0.1:55526 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 01:12:24.243170 23348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.239197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.0.97.61:41885)
0619 01:12:24.239345 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.239349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.241842 (+ 2493us) server_negotiation.cc:407] Connection header received
0619 01:12:24.242061 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.242065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.242137 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.242246 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.242956 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55542: BlockingRecv error: recv got EOF from 127.0.0.1:55542 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 01:12:24.255232 23348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.248186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.0.97.61:41885)
0619 01:12:24.248407 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.248416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.253834 (+ 5418us) server_negotiation.cc:407] Connection header received
0619 01:12:24.254075 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.254079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.254156 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.254268 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.254999 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55552: BlockingRecv error: recv got EOF from 127.0.0.1:55552 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 01:12:24.269347 23348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.267076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.0.97.61:41885)
0619 01:12:24.267361 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.267370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.267891 (+ 521us) server_negotiation.cc:407] Connection header received
0619 01:12:24.268099 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.268106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.268188 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.268296 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.269084 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55558: BlockingRecv error: recv got EOF from 127.0.0.1:55558 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
W20260619 01:12:24.282956 23348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.280693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.0.97.61:41885)
0619 01:12:24.280947 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.280956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.281547 (+ 591us) server_negotiation.cc:407] Connection header received
0619 01:12:24.281839 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.281844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.281929 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.282039 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.282730 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55574: BlockingRecv error: recv got EOF from 127.0.0.1:55574 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 01:12:24.305339 23348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.303022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.0.97.61:41885)
0619 01:12:24.303302 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.303312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.303876 (+ 564us) server_negotiation.cc:407] Connection header received
0619 01:12:24.304110 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.304114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.304203 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.304325 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.305072 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55590: BlockingRecv error: recv got EOF from 127.0.0.1:55590 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 01:12:24.346274 23348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.343731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55594 (local address 127.0.97.61:41885)
0619 01:12:24.343928 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.343935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.344503 (+ 568us) server_negotiation.cc:407] Connection header received
0619 01:12:24.344847 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.344853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.344949 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.345059 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.346010 (+ 951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55594: BlockingRecv error: recv got EOF from 127.0.0.1:55594 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 01:12:24.416025 23348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.414020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.0.97.61:41885)
0619 01:12:24.414246 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.414253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.414650 (+ 397us) server_negotiation.cc:407] Connection header received
0619 01:12:24.414833 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.414840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.414943 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.415073 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.415810 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55600: BlockingRecv error: recv got EOF from 127.0.0.1:55600 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 01:12:24.549811 23348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.547885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.0.97.61:41885)
0619 01:12:24.548149 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.548163 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.548297 (+ 134us) server_negotiation.cc:407] Connection header received
0619 01:12:24.548476 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.548480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.548554 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.548646 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.549557 (+ 911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55606: BlockingRecv error: recv got EOF from 127.0.0.1:55606 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 01:12:24.791316 23436 catalog_manager.cc:1259] Loaded cluster ID: 4c493cb25c1c45ecba625346b813f4b8
I20260619 01:12:24.791450 23436 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d: loading cluster ID for follower catalog manager: success
I20260619 01:12:24.794093 23436 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d: acquiring CA information for follower catalog manager: success
I20260619 01:12:24.795994 23436 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 842e4c132baf444d9c36a703a967e52d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 01:12:24.812047 23348 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:12:24.810143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.0.97.61:41885)
0619 01:12:24.810391 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:24.810398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:24.810631 (+ 233us) server_negotiation.cc:407] Connection header received
0619 01:12:24.810808 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:24.810812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:24.810896 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:24.811051 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:12:24.811836 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55616: BlockingRecv error: recv got EOF from 127.0.0.1:55616 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 01:12:25.100039 23483 catalog_manager.cc:1259] Loaded cluster ID: 4c493cb25c1c45ecba625346b813f4b8
I20260619 01:12:25.100152 23483 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193: loading cluster ID for follower catalog manager: success
I20260619 01:12:25.102813 23483 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193: acquiring CA information for follower catalog manager: success
I20260619 01:12:25.104660 23483 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 217f7bc0df244612b2bb21f453ede193: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 01:12:25.337599 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:25.331254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.0.97.61:41885)
0619 01:12:25.332022 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:25.332028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:25.332042 (+ 14us) server_negotiation.cc:407] Connection header received
0619 01:12:25.332099 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:25.332102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:25.332256 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:25.332415 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:25.332868 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:25.333808 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:25.335146 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:25.335619 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:25.335774 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:25.336284 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:25.336898 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:25.337073 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:25.337136 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 01:12:25.337214 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":448,"threads_started":1}
I20260619 01:12:25.353164 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:25.346422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.0.97.60:41227)
0619 01:12:25.347198 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:25.347203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:25.347222 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:12:25.347323 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:25.347327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:25.347473 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:25.347626 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:25.348122 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:25.348824 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:25.350836 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:25.351300 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:25.351420 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:25.351965 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:25.352524 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:25.352631 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:25.352783 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 01:12:25.352847 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":576,"threads_started":1}
I20260619 01:12:25.377727 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:25.359623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.0.97.61:41885)
0619 01:12:25.363713 (+ 4090us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:25.363723 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:25.370371 (+ 6648us) server_negotiation.cc:407] Connection header received
0619 01:12:25.370819 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:25.370824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:25.370921 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:25.370992 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:25.371849 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:25.372984 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:25.375256 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:25.375971 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:25.376077 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:25.376809 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:25.376951 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:25.377019 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:25.377271 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 01:12:25.377389 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3876}
I20260619 01:12:25.381350 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:25.359332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.0.97.62:32779)
0619 01:12:25.370032 (+ 10700us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:25.370038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:25.372029 (+ 1991us) server_negotiation.cc:407] Connection header received
0619 01:12:25.372279 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:25.372302 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:25.372519 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:25.372715 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:25.373502 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:25.375006 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:25.377833 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:25.378627 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:25.378759 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:25.379991 (+ 1232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:25.380472 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:25.380621 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:25.380724 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 01:12:25.380871 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10549,"thread_start_us":11068,"threads_started":1}
I20260619 01:12:25.385221 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:12:25.366180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.0.97.60:41227)
0619 01:12:25.366408 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 01:12:25.366417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:12:25.378841 (+ 12424us) server_negotiation.cc:407] Connection header received
0619 01:12:25.379010 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:12:25.379014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:12:25.379105 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:12:25.379195 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:12:25.379845 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:25.380948 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:25.383294 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:12:25.383802 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:12:25.383899 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:12:25.384450 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:12:25.384571 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:12:25.384640 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 01:12:25.384848 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 01:12:25.384929 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 01:12:25.388862 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 23203
I20260619 01:12:25.389849 23265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:12:25.537531 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 23203
I20260619 01:12:25.559154 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 23277
I20260619 01:12:25.560267 23339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:12:25.669425 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 23277
I20260619 01:12:25.689939 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskbdbgje/build/asan/bin/kudu pid 23351
I20260619 01:12:25.691063 23413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:12:25.798243 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbdbgje/build/asan/bin/kudu with pid 23351
2026-06-19T01:12:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4984 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9415 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (374468 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST